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PREFACE 



The study herein reported vas in many vays an experiment, yet it 
cannot he called an experimental research study in the ususal sense 
of that phrase. It was by design a product-oriented study and there- 
fore it leans toward the "development" end of the research-and '•devel- 
opment continuum. Neverthless, as the study proceeded it was greatly 
influenced by previously conducted research and the willingness of a 
number of researchers who had conducted that research to work with the 
project staff in interpreting their findings in li^t of the task at 
hand — the development of an effective means of exchanging educational 
product information. A comprehensive list of all the members of the 
research community who in some way contributed to the work described 
in this report would be lengthy, and might appear to be an argumentum 
ad homines for the worth of the work described herein. However, there 
are a ' fe Tresearchers whose contribution to the development of the de- 
sign of the product information exchange, as it evolved in the course 
of this study, has been such that it would be impossible to fail to 
mention them individually. They are: 



Dr. Terry Denny, now of the EPIE staff but formerly Associate 
Professor of Education and Psychology, Purdue University 
Dr. J. A. Easley, Jr., Associate Professor of Education, Uni- 
versity of Illinois 

Dr. Robert T. Filep, Education Systems Project, System Developineno 

Corporation , ^ 4.4 

Dr. Frederick L. Goodman, Associate Professor, School of Education, 

University of Michigan ^ ^ „ 

Dr. Ira J. Gordon, Director, Institute for Development of Human 
Resources, College of Education, University of Florida 
Dr. Robert E. Stake, Associate Director, Center for Instructional 
Research and Curriculum Evaluatipn, College of Education, 
University of Illinois 

Dr. Raymond Wyman, Professor of Education, Audiovisual Center, 
University pf Massachusetts 

In addition to the work of these individuals the project has had 
the benefit of direct assistance and a general willingness to cooperate 
from a number of professional associations. Particular mention and 
gratitude are due the Commission on Technical Standards of the depart- 
ment of Audiovisual Instruction, NEA, and the Commission Instrucoion^. 
Theory of the Association for Supervision and Curriculum Development. The 
work of these commissions has had considerable influence on the study, 
through the advice and counsel of their respective chairmen, Drs. Wyman 

and Gordon. 

The Director also wishes to acknowledge his debt to Dr. Joseph L. 
Dionne, v’no helped organize the study in its early stages hut «Jo left 
the project staff soon after the study began. Members of the staff 
uho rarried out the bulk of the "in house" vork, and to whom the Director 
is particularly indebted, are Mrs. Anne Abramovitz, Mr. Kenneth E. Baranski, 

and Miss Bett’vr Braxton Preston. 
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SUMMARY 



School decision-makers and designers and devisors of instruc- 
tional materials, systems, and equipment are in need of systematically 
collected and disseminated descriptive, analytipal, and performance 
data on such materials, systems, and equipment. In the course of the 
study herein described, the Educational Products Information Exchange 
(EPIE) Institute has planned and outlined a system for the collection, 
synthesis, and dissemination of such information. 

Plans and procedures include meetings between users and producers 
Qf equipment to discuss what characteristics of hardware need be 
described to permit informed selection. Professionals at two univer- 
sities have devised procedures for analyzing content and explicating 
the pedagogical assumptions underlying the make-up of instructional 
materials. Interview protocols for use with teachers have been devised 
and tested, as have methods for training school personnel to use EPIE 
information collection techniques. A pilot study of the entire system 
has been designed, and a preliminary version of a comprehensive systems 
design for the Exchange has been completed and is being revised in 
preparation for the proposed pilot study. 

The Exchange is designed to develop publicly and in the public 
interest, and to call upon the assistance of educational professionals 
in its development and operation. Plans for service include broadcast 
of general information throu^ publications and, eventually, responses 
to ad hoc questions and tailor-made service for specific needs. Sub- 
scription rates to make services self-supporting will be developed. 



INTRODUCTION 



New media, the "systems approach" to education, new ter^» hooks, 
innovative techniques, new curricula, more Federal Funds — all 
these and more factors have come, in recent years, to increase the 
school decision-maker *s hopes for the optimum pupil-material match, 
as they have concurrently reduced the possibility of the decision- 
maker's being able to look at all available materials before trying 
to make that match. 

The same factors, of course, have affected the work of the 
producer of educational materials, who is supplying that most chal- 
lenging market in educational history without accurate, lndlva.duallzed 
knowledge as to what users want or how available products achieve or 
fall short of success. 

It has, therefore, become increasingly apparent that schools and 
producers alike are in pressing need of effectively organized, accurate 
accessible information about educational materials, equipment, and 
systems, and that the use of such information in the selection, use, 
and design of such products would have a salutary effect on educational 

practice . 

The present study* was undertaken for the purpose of designing 
an easily accessible nationwide system for exchanging descriptive, 
evaluative product information among all sectors of the educational 
community, on a cooperative cost-sharing basis. Educational pro- 
fessionals in schools and industry will be encouraged to contribute 
to and use this product information exchange as a regular part of their 
responsibilities, thus not only improving the basis for their own pro- 
fessional judgments but also contributing to the improvement of edu- 
cational technology in general. 

The designers of this exchange approached their task with the 
belief that their research must produce the design of a service that 
will function in the public interest, utilizing the assistance, ad- 
vice, and special competencies of many cooperating professionals. 
Furthermore, it must be designed so that after its developmental 
period, it will be supported completely by those who make use of its 
service. It must be responsive to the needs of .its users, and must 
guarantee a carefully developed and clearly defined base for the data 
which it reports. It must not only take due notice of new product 
developments, but must anticipate them and seek out early reports of 
them. 



W should be pointed out that the Educational Prod its Information 
Exchange (EPIE) Institute, the contractor for the present study, has 
operated independently only since August 1, 19^7^ Prior to that time 
it existed as a division of the Institute for Educational Development, 
and the beginning of this study was undertaken there, under an earlier 

contract. 
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The builders of the exchange recognize that the great variety 
of educational purposes and practices makes it impossible, even if 
it were desirable, to "rank" or evaluate educational ® 

universal sense; what works well in one educational setting with one 
set of students will not necessarily work well or be deemed suitable 
elsewhere. What works poorly in a first marketed version ^ 
greatly improved after revision. Therefore the most useful informa- 
tion will be reports of systematically sampled users as to the per- 
formance of a product in specific institutional settings over time, 
synthesized with the producer *s description of his product and a^lyses 
of the product by independent researchers. These are the three types 
of product information the present study has shown to be useful and 
feasible to gather, synthesize, and disseminate, ^e study also 
has explored the development and preliminary use of specific techniques 
for identifying and processing these three basic types of product in- 
formation. 




THE ATTACK ON THE PROBUBM 



The Enlistment of the Aid of the Professional Community, 



The EPIB Advisory Board 

The designers of the educational products information exchange 
saw as the first order of husiness the establishment of continuing 
cooperative arrangements with institutions and agencies engaged in 
relevant research, associations of educators, and associations of 
producers. Visits to and correspondence with such organizations 
were followed by a series of exploratory meetings for groups whose 
members had- shown or might be expected to show interest in the 
exchange’s aims and development. 

The first of the series of group meetings came in ®®rly 
December, 1966 , when representatives of educational administration 
organizations met to hear about, react to, and help shape the develop- 
ment of the exchange (by then Imown as EPIE, its acronym). They 
suggested that representatives of their and other appropriate asso- 
ciations be constituted as an EPIE Advisory Board, which would con- 
tinue to play a role in the development of the exchange. Electors 
and executive secretaries of teacher associations, meeting the next 
month (January 196 ?), asked to be included on the 
Representatives of producer associations, also meeting that ®°ath,. 
were interested in hearing of EPIE’s progress as it developed, but 
were, for the most part, not enthusiastic about direct involvment 

in its development. 

Thirty tJiiociatloxiB . (listed as A) now ^ -u j. „„ j 

the Ae^^y Board which has had two formal meetings hear about and 
advisfon dLlopments. Between meetings individual representatives 
have cooperated with EPIE in a variety of ways, ranging from Weuti- 
fication of key personnel who rai^t take an active part in EPIE 
development to cooperation in conducting meetings and training 

sessions. 

One of the Advisory Board’s most important function will be the 
nomination of members for professional auditing commiees, w 
least annually will review EPIE’s procedures and practices, thus 
profidi^an s^rce of evaluation and guidance in specific 

endeavors . 

EPIB’s Consultants 

During its development EKE has received the substantive help of 
a diverse group of consultants, some of whom are connected with co- 
operating associations and others of whom have been asked or have 
volunteered to cooperate because of the peculiar relevance of their 







■work and interest to EPIE's purposes. In the course of the -nresent 
study consultants have not only advised on overall planning ut, as 
subsequent sections will show, have developed preliminary systems 
designs, run initial training courses for information gatherers, 
chaired cooperatively sponsored meetings, devised experimental 
questionnaires and interview protocols, devised methods for analyzing 
instructional materials, and in general supplemented and augmented 
the work of the small central staff. 

Some thirty consultants, who volunteered their time, met for 
three days in December, 1966, to discuss in detail the sorts of in- 
formation EPIE must collect in order to be most useful, and how that 
information mig^t best be collected. It became increasingly obvious 
that EPIE must, in order to develop the soundest possible procedures 
to insure the dependability of its information, try its collection 
procedures, its analysis techniques, its operational definitions, and 
its dissemination system in a series of pilot studies, and plans were 
set in motion to devise such studies. 

The Definition of Useful Product Information Types and Services 

Examination of currently a^/a liable product information, discussion 
of selection practices with school personnel, and the experience-based 
advice of the consul'bants who discussed the matter in detail at the 
December work conference bore out the contention on which EPIE was 
founded that systematically organized information giving comparable 
data on similar products was a pressing need among school decision- 
makers, The same sources corroborated the need for performance data, 
which the founders of EPIE had foreseen, and emphasized the usefulness 
such information would have for producers as well as for school selec- 
tion personnel. They emphasized, too, the necess^ ’.y for relating per- 
formance information to the instructional setting, pointing out that 
EPIE could make a valuable contribution to education by establishing 
empirically the role of various factors known and suspected to have 
effects on the performance of instructional materials. Finally, these 
same sources made clear the need for another kind of information: 
independent explications of the pedagogical philosophy and assumptions 
about students, teachers, and the subject matter which are inherent 
in the content and structure of instructional materials. 

Repeated discussion of the needs EPIE planned to fill and of the 
steps to be taken to do so made it clear that helpful information 
would be in hand and should be made available long before the full 
measure of service can be developed. It was equally clear that in 
certain situations the full measure of service would not be called 
upon, a point which strengthened the determination to offer general 
information early in EPIE*s career and to add various levels of ad 
hoc information as development permitted. Attached charts. Figures 
1 and 2 , chow -information and sei*vice schemes. 
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Types of Educational Product Information to Be Collected by 

The Educational Products Information Exchange r 


Type 


, 

1 

i 

Sources j 


Product Producer I 

Information 1 


! a) Producers* catalogues, reports of 

j surveys and validation studies, | 

promotion pieces, advertisements, etc# , 




b) Direct contact with the producer, in 
meetings or by questionnaire. 


l__ - 

( 

' Product Analyst 

Information 


i 

! a) Reports of independent scholars and 

• researchers in universities, regional 

laboratories, research and development 
centers, state education departments, 
etc. who will be commissioned by 

1 ERIE. 




» 

j b) Professional Journals, monographs, con- • 

i ference proceedings, etc. 

— ^ — — — ■ ■ ' - - — — — J 



Product User 
Information 



a) Inventories and questionnaire responses » 
of teachers, students, administrators, • 
and parents surveyed by EPIE. 



b) Classroom observations undertaken by 
EPIE. 

c) Reports of the results, rating, scale 
results, etc,, solicited by EPIE. 



I 

J 




o 
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Modes of Subscriber Service to be Offered by 
The Educational Products Information Exchange 


1 

! Mode of Service 


Degree of Ser- 
vice’s Focus on 
the Subscriber 


Begin First 
Level of Service 


Service Fully 
Operational 


1 ■ " 

j I. Broadcast 
1 Mode: trans- 

1 mission of 

i generalized 

\ product 

information 

f 


Diffuse, 

incidental 

1 


September 

1967 

1 

1 

1 

i 

i 

! 


September 

1969 

« 


} 

’ II. Responsive Marginal 

Mode: acting definition 

j on standard 

^ inquiries with 

data in system 


December 

1968 


January 

1970 


III. Interactive Near maximal 

Mode: dialogue definition 

between 
EPIE and 
subscriber be- 
yond standard 1 

■ inquiry proto- 

i cols : 

1 

■ 1 - - , 


itov ' January 

I9S9 ! 1971 

i 

1 

1 

'■ 1 

! 

i 

1 ! 


1 IV. Customized Mdde: j Adjust until 

tailoring system highly focused 

j and field studies, ’ 

! if necessary, to | 

j gather or exchange - 

i information not j 

provided by Ser- 
vice Modes I, II, ' 

III 1 ■ 


September 

1970 


January i 

1971 I 

{ 

\ 

t 

* 1 

\ 

j 1 
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The Establishment of Limits for the First Pilot Study 



Several factors worked together to set the various necessary 
limits for the first pilot study. The Eastern Regional Institute for 
Education, a regional education laboratory which operates in New 
York State and part of Pennsylvania, had offered help f r m EPIE's 
inception, and on its advice it was decided to conduct ti -i study in 
four states jointly served by that and another regional laboratory. 
Research for Better Schools, Inc., active in the rest of Pennsylvania 
and in New Jersey and Delaware. The encouraging cooperation received 
from the two laboratories was matched by the state education authorities 
in all four states, who agreed to petition the U.S. Office of Education, 
under Title V of the Elementary and Secondary Education Act, for funds 
to conduct the first pilot study within their borders.* 

The determination of curriculum area and level limits was based on 
the results of a study conducted in the fall of 1966 by the Institute 
for Educational Development to discover which product classes were of 
immediate interest to school decision-makers. The staff investigated 
various market reports dealing with projections of what schools would 
buy in the immediate future, and compilations of sales figures for 
recent years. More importantly, and with more clear cut results, the 
Institute for Educational Development sent questionnaires to school 
superintendents (nominated for the purpose by the Executive Secretary 
of the American Association of School administrators) all over the 
country to inquire as to their immediate interests. This survey indi- 
cated greatest interest in the reading curriculum and in equipment for 
using videotape. Unwilling in its first trial to take on such very 
complex aspects of teaching, EPIE moved to the somewhat more manageable 
"runners-up" in the Institute for Educational Development's survey; 
elementary school science materials and overhead projectors. 

The Design of a Data System for the Exchange of Information 



In the course of the December work conference, general attention 
was given to the design of an information storage and retrieval system 
for EPIE by all the consultants present and in particular by the repre- 
sentatives of the Community Systems Foundation, Ann Arbor, Michigan, 
with whom EPIE had contracted for a preliminary statement of such a 
design. Tlie desJ*gn formed part of the subject matter at a subsequent 
consultants' work conference, in April of 19 o 7 > and in late April the 
Community Systems Foundation submitted to EPIE the document attached 
as Appendix Br Experience over the summer indicates that modifications 
in the design will be necessary; Appendix C outlines a broader concept 
which more nearly reflects the present intentions for EPIE's develop- 
ment over the next three years. 



^Changes in and debate on the initially proposed bill to extend the 
Elementary and Secondary Education Act have so far prevented any 
implementation of this proposal. 
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The primary aim of the information storage and retrieval system 
design for EPIE is not noticeably different from that for any such 
design— the rapid and efficient entry of new data into the system 
and the rapid and efficient retrieval of data relevant to an input 
request* It is expected, however, that input requests may be very 
complex, and special attention must be given to retrieval software 
to accommodate them. Also, much attention must be given to the 
selection for synthesis of the most up-to-date information of all 
three types on any given product, with earlier information retained 
for recall if needed. Still further calls on the system vill have 
to do with investigation of what stored information is being used, 
including the entry of feed-back information on its perceived effective- 
ness. 



It should be noted that present ideas about an information storage 
and retrieval system for EPIE indicate the desirability of arranging 
for shared time in existing computers— perhaps ones located in centers 
vhere activities related to EPIE are going on— rather than the establish- 
ment of an EPIE computer complex,. Further experience with EPIE, together 
with the inevitable computer improvements during the projected three- 
year development period may change the thinking on this matter. 

The Development of Information-Gathering Techniques* 

Assembling Producer Information 

It became obvious as EPIE began to collect information on over- 
head projectors, the equipment class to be reported on first, that 
some correlation of availability of and need for information was called 
for. Accordingly EPIE, working with the Department of Audiovisual 
Instruction of the National Education Association (a member of the EPIE 
Advisory Board), drew up an exhaustive list, containing every character- 
istic an overhead projector can be imagined to have, and sent it to 
representative users of projectors and to all producers of projectors, 
with the request that they judge the importance of each of the one 
hundred or more characteristics. The results were discussed at a meet- 
ing jointly sponsored by the Department of Audiovisual Instruction and 
EPIE, at which representatives of the user group and of the producer 
group reviewed each characteristic. The discussion resulted in a 
"priority list" of the characteristics deemed important by the repre- 
sentatives of both groups, as well as agreement as to the necessity for 
supplying comparable information on similar products. The priority 



*Research and development in these techniques were only partially 
supported by U.S. Office of Education funds. Other supporting agencies 
were the Pennsylvania Department of Public Instruction, the Institute 
for Educational Development, and the Universities of Florida and Illinois. 



list makes it possible for EPIE to use producers* published informa- 
tion selectively in compiling information on overhead projectors and 
also gives producers a mechanism for reporting on new projectors* A 
group of school audiovisual personnel will be asked to comment on the 
usefulness of the descriptive chart devised on the basis of the 
priority list, with a view to refining the mechanism as use dictates. 
Plans have been laid for a similar attack on the question of describ- 
ing closed circuit television equipment. 

The question of priorities of information in regard to educational 
materials of a content nature is soraehwat less complicated, since matters 
of precise electric and other such technical measurements do not occur. 
Informal discussions with several experienced science teachers have re- 
sulted in some guidelines for the initial collation of producers* 
descriptive information on elementary school science mater j^ls. Further 
such discussion and submission of the resulting lists to use tests will 
be necessary here also. 

Developing Analyst Info .nnat ion 

As indicated earlier, the consultants who met at the December and 
the April work conferences addressed themselves in detail to the question 
of independent analysis of instructional materials for the purpose of 
explicating the assumptions which underlie their preparation. At the 
second work conference attention was given to various approaches to such 
analysis, and since that time groups working at the Universities of 
Florida and Illinois have undertaken experimentally the development of 
practicable analysis techniques. Appendices D, E, and F relate in more 
detail the results of the work conferences and of experimental trials. 

The value of the planned product analyses may go beyond their role 
in the selection process. VThile their first usefulness is perhaps to 
permit the school decision-maker to discover which products fit most 
closely the purposes, procedures, and educational philosophy which 
serve as his teaching framework, secondarily they may call to his 
attention, for instance, another approach which may add a new dimension 
to his program. Detailed analyses, too, should facilitate the process 
of integrating instruction from year to year by permitting the school 
to select materials which fit together. Finally, the analyses will, 
as a number of producers have already indicated to EPIE, be very useful 
in design and revision of materials. 

Collecting User Information 

From the beginning, EPIE has planned to include in its descrip- 
tions of products reports about user satisfaction and dissatisfaction. 
Many different product dimensions and conditions of product use will be 
taken into account in collecting the information and in reporting it to 
subscribers. 
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Reports on the perforn>ance of produots ip relation to specific 
instructional settings will he gathered separately from teachers > 
administrators, and students* Interview response forms will he 
tailored to fit these three sources* The first such form> that for 
teachers, has been drawn up and is currently being tested on a small 
group (N*^) of elementary science teachers* (See Appendix G#) 

Interview reports will he supplemented by questionnaire and tele^ 
phone » urvey techniques* The collection of data will for the most part 
he undertaken locally by school people who have had considerable experi- 
ence in educational research or who have had specific instruction in 
ERIE information-^thering techniques at a rather extensive training 
session* The first such session was held between August 13 and 31, 

1967, to test the training strategy. A report on that session appears 

as Appendix E* 

In seeking balanced product performance information, ERIE must 
insure that it has been in touch vith a directly describable sample of 
product users* A scheme for selecting such a sample for the first 
pilot study, the Pour-State Cooperative Pr object, is attached as 
Appendix I. 
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^mSRE THB EXCHANGE SZANSS 



Th6 present study has yislded results iiiilch indicate that the 
collection^ syntheslsi and dissemination of educational ixroduct in- 
formation is a feasihle activity. There are, Indeed, Indlcatjons that 
findings incidental to the pursuit of that activity vlll he of Interest 
and use to the educational field in gen^^al. Appendices J and K 
es^Ucate in detail the nature and scope of the Institute’s idnns for 
research and service. 

In the course of the study EFIB has established professional 
connections vhich vlll he Invalvahle to its ;future dev^pment. More 
Importantly, it has evoked in a large segnent of the educational cem- 
raunlty a response vhich in effect demands that the activity he made 
operational. Some, thoucii not aU, of the teehnloues needed to effect 
the collection of pertinent information have had sufficient trial to 
he considered final; starts have been mado at least in all the necessary 
avenues, and comprehensive plans are laid down for continuance. The 
present uncerti^ state of funding for the Four-State Crooperative 
Project is the only deterrent to Inmedlate Implementation of the full- 
scale research program. 



THE FUTURE 



As the foregoing has indicated, emphasis so far in EPlE*s 
development has been on the collection rather than the dissemination 
of information. Broadc^ist service, the first 

for EPIE, is possible now, and has been initiated, but not under this 
contract; expansion to responsive and interactive service is schedul 
to take place during the proposed three-year development period, as 
is the addition, if it is in demand, of a customized service, 
actually tailor-made to the requester's specifications, Suph develop- 
ments, of course, depend upon successful trials of collection and 

synthesis techniques, as well as upon another ^ ^^® 

the development of the services in such a way that they necome self- 

supporting after a period of trial and development. 

A philosophical aim of EPIE’s staff is 
provement of rather thrn just change in educational technology. EPffi 
feels a responsibility to work towards the humanistic use of the toolb 
and the products of the increasingly technological education Profession. 
EPIE very much hopes to develop its services in such a way as to facili- 
tate education of individuals rather than of groups. 
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LOI^G«»RAiNGS GOALS Al^ THSIil ACHISVEI-lSMT 



In designing and implementing an information system, it is often 
beneficial to consider those characteristics which represent a system that 
fully achieves all desireable goals* In so doing, the creators of the 
system force themselves to state in precise terms the ultimate objectives 
of their work. 

It is important for those who will guide SPIE*s future to formulate 
a clear statement in operational terms of what EPIE seeks to achieve and 
what barriers must be overcome to do so. For EPIE will be created with the 
recognition that it cannot at the outset fulfill all of its goals, yet must 
continually progress closer to them. 

For many reasons, EPIE cannot at the outset achieve all of its 
objectives. To do so, EPIE would have to exhibit at least the following 
characteri sties: 

1, Users of the system would have the ability to gain access to 
information which would always influence the attainment of a 
"correct” decision for product selection and application, 

2, ilultiple users of the system would have the ability to achieve 
simultaneously instantaneous access to information as frequent^ 
ly as desired, 

3, The system would continuously experience full utilization of 
its resources, including personnel, plant, and information, 

4, The cost of operating and maintaining the system would be zero. 
Underlying the characteristics above are a number of sub-goals which would 
have to be ?iet and yet jannot be at the outset. 

To achieve the characteristic of "correct" information, objective 
measures would have to be available to determine the "correctness" of each 

* IT - 



decision arrived at by users of the EPIB system. Should a decision be 
judged correct on the basis of student performance using the chosen materlalsT 
On the basis of teacher's accepfance of the chosen materials? On the basis 
of whether or not the system user continues to use the EPIE system after a 
personal evaluation of the effectiveness of past use? Should It be judged 
correct on the basis of one, all, or a combination of these texts? Is one 
test more Important than another? If so, how should each be weighed tela- 
tlve to the others? Within a test, how should terms such as "performance" 
and "acceptance" be scaled? Further, Is student performance computed on 
the basis of percentile scores on standardised tests administered at the 
completion of a course In which the chosen materials were used, on the basis 
of the stature achieved In adulthopd by the student In the study area, or 
by some other measure? How can the contributions to student perUrmance 
made by the selected materials be Isolated from those deriving from other 
materials, or from the teacher's role, or from exposure to other students? 

One could continue to ask such questions, with each serving to sub- 
stantiate the difficulty of determining the "correctness" of a decision. 

Yet in an Ideal system, the meaning of "correctness" would be clear and 

universally accepted* 

Operationally, the EPIE system can go a long way towards the Isola- 
tion of those factors which appear to account for judgments as to the 
correctness of decisions. Statistical techniques can be applied which help 
to explain differences In judgments as to the correctness of a decision or 
the utility of a material. However, the successful use of these techniques 
does not rest on the ability ^f EPIE to attract skilled statisticians who 
can perform meaningful analyses of variance. Rather It rests on the ability 
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of educators to 1) identify the factors or variables which do in fact cause 
the judgments of one person to vary from another and which cause the per- 
formance of one student to vary from an"'ther$ 2) define these variables in 
operational terms which are understood and accepted by those who must use 
them, 3) design techniques for measuring ^d scaling the variables which are 
easily applied, acceptable to those performing the measurement and scaling, 
and valid with respect to possessing the sensitivity essential to achieving 
the explanation of variance which is Inherent in the judgment being made or 
the performance being measured* 

Months of work by leading educators in the fields of curriculum ana- 
lysis and performance measurement have shown that the above capabilities do 
not exist today. Progress has only just begun to be achieved in the above 
activities. Without these capabilities, the skills of the statistician in 
performing analyses of variance, the results of which would be ussd to 
synthesize data in the preparation of information for use by EPIE users- 

decision makers, would be of little value* 

To achieve the second ideal characteristic above — Instantaneous, 
unlimited access to information within EPIE — several requirements would 
have to be met which either exceed the resources likely to be available in 
the near future or which cannot be achieved prior to the gaining of exper- 
ience via actual operation of the system. Implicit in the ability of the 
system user to achieve access to information whenever desired are the re- 
quirements discussed above under the first ideal characteristic, plus the 
corollary characteristic of EPIE possessing the pre-requisite resources for 
fulfilling user demands* Fundamental to possessing such capability within 
EPIE is the possession of full knowledge of the demands. .which will be Imposud 
upon the EPIE system, sufficiently early to permit the planning and 
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acquisition of resources. Such knowledge would consist of demand sources, 
frequency of inquiry, processing time requirements (workload), user response 
time requirements, user location, user requirements for information display, 
etc. 

Although survey techniques exist for forecasting demand and although 
EPIE has gained some knowWge as to potential sources of demand by defining 
a geographic pilot region, little can be done to forecast demand in the pre- 
cise terms required for ideal planning and resource acquisition. Thus, EPIE 
must initially forego the capability of meeting potential demand 
instantaneously. Rather, it will saci^lflce servicing a portion of the demand 
lack the capability of providing "on-demand” service, or both. 

Even if EPIE could forecast total demand in the terms required, this 
would not be enough to achieve the ideal characteristics of Instantaneous, 
unlimited access to Infonnatlon. To achieve such access to EPIE information, 
fully automated information managei»ient and retrieval is pre-requisite. 

The fulfillment of the first Ideal characteristic will ultimately 
create a massive amount of infonnatlon requiring repetitive processing in 
view of specific Inquiries. A highly sophisticated Infonnation system of 
the type ultimately envisioned will require large amounts of data to achieve 
acceptable statistical significance in its analyses of variance. Rather, 
it would also be designed to permit, when needed, the performance of such 
analyses. This will require the storage of large amounts of data in a 
form amenable to rapid search, retrieval, and manipulation, and the pre-pro- 
grar.iming of routines or rules for processing t)ie data. Such tasks are 
ideally performed with the use of high-speed elqptronic computers, as they 
represent the most economical and reliable means for repetitive manipulation 
of large amounts of data in "random" or unpredictable c-^mbi nations. 



For economic reasons^ automation will be required if system users 
are to achieve instantaneous access to information as needed* Under such 
conditions, the system must anticipate the presence of multiple inquiries 
which simultaneously require the use of one or several sets of data* Under 
a manual or partially automated system, this requiarement could only be met 
by duplicating data and data processing staff in sufficient quantity to avoid 
the unacceptable delay of one user while another’s inquiry is being processed* 
However, through the use of today?! high speed, "time sharing*' computer 

•t 

systems, the need to maintain duplicate sets of data and staff for inquiry 
processing can be eliminated* Such systems permit many users to simultan- 
eously share one set of data and staff with the effective appearance of 
exclusive use* 

However, the initial EPIE system is not likely to possess such 
capability* For one thing, it would be difficult to meet the computer pro- 
gramming requirements within the time and resources likely to be available* 
For another, the amount of data initially available, the volume of initial 
demand, and the complexity of the initial analysis-retrieval procedures are 
not likely to justify the use of computers* Also, the acquisition of the 
necessary computer hardware, including communication devices for use in re- 
ceiving, processing, and returning the results of system user inquiries 
could not be achieved in the initial pilot period* Also, the training of 
users in how to communicate with such a system is a large task requiring the 
preparation of detailed instructional materials and considerable training 
time* 

Thus, EPIE for a number of reasons cannot initially possess the 
characteristic of full (integrated) automation* However, this must be a 
major goal of the system if reliable, "personalized" information synthesis 
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and ‘'on-demand** sen^ice to users are to be realized. The challenge to EPIE 
will be to transcend from a state permitting the servicing of individual 
inquiries over several '’ays, weeks, or even months to a state wherein 
curriculum committees and others can utilize the EPIE system "live** as they 
perform the tasks of educational product evaluation and selection. 

Finally, the ideal system characteristics of full resource utilization 
and zero cost cannot be met in an operational EPIE, no matter how sophisti- 
cated the system becomes. Yet these ideal characteristics should represent 
goals whose achievement is constantly being sought. Full utilization of 
existing resources should represent a guiding principle of never acquiring 
resources which are idle to a substantial degree. Zero cost, in turn, 
should stand for the principle of cost minimization through effective 
organization, planning, scheduling, methods selection, and supervision. 

SHQRT«RUM GOALS 

Although EPIE cannot achieve all of its long-range goals at the 
outset, it can be of service to educational decision makers in the selection 
of materials if it can partially fulfill these goals. The task at hand is 
to design a meaningful initial operationa_l system. The term ''operational 
systera** is intended to represent a particular information system which 
provides limited but useable information and service, which passes the tests 
of economic justification and survival, which recognizes in highly specific 
terms how it falls short of its long-range goals, and which continuously 
strives to achieve them through evaluation, research, revision, and sound 

management. 

Unquestionably, there exist as many operational systems as there are 
individuals or groups to design such systems. The recognition of the need 
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to d'isign a less than "perfect” system, creates an opportunity to design a 
number of systems which cannot je evaluated against benchmarks of perfect 
performaiice. It Is difficult to evaluate the superiority of one "sub-optimal" 
design over another. As such, the operational system which EPIE will 
Initially achieve should not be taker, to represent the one and only opera- 
tional system. Rather It will be a system which, hopefulljj will adequately 
meet EPIE»s short-range goals, while possessing the Important "Ingredients" 
for long-range goal achievement. 

Perhaps the need for creating an operational system should be stressed. 
From the viewpoint of those members of the education profession who have seen 
In EPIS a means of vastly Improving the decision-making process of educa- 
tional materials selev 4on, an overriding need exists now to create an oper- 
ational EPIE which will be justified as long as It provides the decision 
makers with better information than they now possess. To this extent, an 
annotated blbliogra hy of all available materials In a particular curriculum 

area would represent an improvement. 

However, If the system was comprised solely of the resources necessary 

to prepare and maintain an annotated bibliography. It would not .neet the 
definition of an operational system as set forth, for It would not Include 
the resources essential to carrying it closer to the long-range goals. 

Those who have supported EPIE are as much committed to an Initial 
system which has the resources to continually evaluate and upgrade itself 
as to one which has resources to serve users on a current basis. To this 
extent, the initial system -called PI1=0T EPIS - should be one part 
service and one part research . That part which Is service will In some ways 
be less Important Initially than the part which is research, for It will be 
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created within fairly severe constraints of limited time, money, and knowledge. 
In the process, it will sacrifice the achievement of mahy of the characteris- 
tics of the long-range system, for Its overriding goal will be a pragmatic 
one of achieving some form of useful service within a short period of time, 
plus effective mechanisms for evaluation and modification as necessary. 

Because, the orientation of the various consultants and advisors to 
EPI£ varies with regard to discipline, experience, working environment, and 
Interests, no uniform opinion exists as to what should or should not be in- 
cluded in PILOT EPIE from a service viewpoint. Some feel a strong need to 
provide initially information pertaining to the cont^t of educational mater- 
ials as well as their physical characteristics and the environments in which 
they have been used in the past. Others feel that concern over content will 
prohibit creating a PILOT EPIE in the near future which provides usefu l in- 
formation. Still others see a major need to execute some semblance of 
statistical analysis in order to synthesize diverse information on a product, 
while others either intuitively believe variance is ultimately unexplainable 
by such techniques or that this element of activity is not essential to pro- 
viding useful information in the early stages of EPIE. Some feel that the 
pilot system can be designed to permit to a limited but meaningful degree the 
dissemination of selected information to individuals, while others doubt the 
practical value of such activity and even fear its consequences. The latter 
group would emphasize the dissemination of generalized information to large 

gxoups of school people. 

However, as the consultants come to realize that PILOT EPIE is an 
experimental system designed to be of some service^ ->ut also designed such 
that mechanisms for evaluation exist which will, permit continual analysis 
of the initial design and subsequent nwdlf Icatlon, they should come to accept 
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the Initial decisions of what will or will not be Included in PILOT EPIE. 

As long as the decisions taken lead to a continuing system, the valid Interests 
of all concerned will eventually be incorporated. 

GENERAL SPECIFICATION GUIDELINES FOR PILOT EPIE 

Introduction 

From the foregoing discussion plus an exposure over many months to 
people concerned With creating a PILOT EPIE, a number of potential system and 
performance characteristics can be identified which may be helpful in guiding 
the initial design. These characteristics and some of the as-yet -unsolved 
issues subsumed in them are presented here. The intent is to suggest 
guidelines and alternatives for detailed design which will aid the designers, 
but not constrain them In the selection of specific policies or in the actual 
design of an operational system. 

Standardized vs. Customized Information Diss emination 

Two major alternative mechanisms exist for conducting the exchange of 
product information. The first, called '‘standardized'* information dissemina- 
tion, would consist of the simultaneous dissemination of a particular sot 
of information to a large group of system users. The information being 
disseminated could be generalized conclusions contained in an analysis of a 
product, generalized conclusions drawn from a number of product evaluations, 
or merely a reprint of an evaluation which draws no general conclusions (e.g.- 
the results of a case study). 

This dissemination woul be achieved by publishing a newsletter 
and special papers which are available to all who are willing to pay a sub- 
scription fee. As such, the extent to which EPIE would control the dissemi- 
nation of information under this approach would be limited to its ability to 
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decide what it will publish in the newsletter. Once it mak s a decision to , 

j 

include a particular set of information^ it will cease to control to any 
extent who has access to the information. 

> 

t 

Alternatively, ’'customized” information dissemination represents an ! 

approach which introduces controls over who gains access to information as i 

well as what information is provided. Whereas a user under the "standardized” | 

approach can gain apcess to any information selected for dissemination j 

I 

simply by passing the "test" of possessing a paid subscription, a user under 
the '"'customized" approach will have to pass a number of additional "tests" in 
order to gain access to such information. Under the "customized" approach, 

! 

certain information available for dissemination may be seen by some system 
users but not others, while under the standardized approach all such informa- 
tion is seen by all system users. 

It is felt that both approaches potentially have merit, although each 
presents problems which are not found in the other or are found to a lesser 
extent. The "standardized" approach recognizes the potential service to 
schools to be achieved in the wide-scale dissemination of knowledge about 
educational products. Those who support this approach believe that exper- 
iences with products can be reported on in such a way as to be of value to 
many schools simultaneously and without pre-screening of those who see the 
information. 

However, it is also possible that in an attempt to keep the published 
information sufficiently generalized to be of interest to at least a large 
portion of the system users (subscribers), several effects may occur. One, 
the reader may not find the information to be detailed or comprehensive 
enough to be of specific value in choosing products for his educational en- 
vironment. Two, the reader may have to read a substantial amount of 



- 26 - 



o 



information not particularly relevant to his immediate needs or interests In 
order to find that which is relevant. Three, the information may be 
sufficiently detailed to be of value to the user, but may not be presented 
in a format which permits him to easily extract those details of personal 
relevance. Four, information which is intended primarily for users with 
particular educational environments may laiwlsely influence decisions by 
users in diffeirent environments. 

It is not kno\m that such effects will occur, nor even hypothesized 
that they will. Since the information retrieval design requirements are 
rather simple under this approach (iec. set up a newsletter subscription 
service), it is generally agreed that it should be incorporated into PILOT 
EPIE for trial and evaluation. 

Under the ’’customized” approach, the information retrieval process 
will be more complex and difficult to design. Two major concerns will be; 

1. Extracting from the system information on only those products 
likely to be of immediate interest and relevance to a system 
user in view of; 

a. His statement of his immediate product selection interests, 
expressed in the form of an inquiry (e.g. ■ Elementary 
Science kits for fifth graders; an Elementary Science 
series, K-6) 

b. His description of the educational environment and condi- 
tions under which the products are to be used (e.g. - cur- 
ricular goals, teacher types, student types, community 
factors, class sizes, etc.) 

2. Providing the system user with product information which is 
organized so as to simplify as much as possible his review and 
analysis of the information. 
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Ihe potential value, of this approach over the "standardized" one Is that 
the user will receive more concise, relevant Information which Is "tailor- 
made** to his Interests and needs. 

However, It is by no >neans proven that such an Information retrieval 
system can be economically designed and operated, or that It Is possible to 
achieve a mote meaningful service to users In terms of providing more concise 
and relevant Information which will lead to bettor decisions than can be 
realized with the "standardized" approach. In order to gain an objective 
comparison of the two approaches, the pilot study will have to Incorporate 
both approaches to Information dissemination and carry out comparative ana- 

lysos based upon experience. 

For EPIE to provide highly customized, exceptlon-oriented Information, 
it must of necessity know more about Its users than It would under the standard 
ised approach. To gain this Inforaiatlon, It w'll be necessary to engage In 
a dialogue with the system users In order to ascertain Important variables 
to be considered In synthesizing Information for feedback to the Inquirer. 
However, no conclusive agreement has been reached on Just what variables must 
be ascertained or what techniques should be used to do so. Some feel that 
highly structured, catch-all questionnaires and check lists should be used. 

The advocates of this technique place little confidence In the ability of the 
inquirer to define on his own volition those variables which are important to 
him and feel as such he must be forcefully guided In his statements of 
relevant Information. Others feel the Inquirer should be permitted to state 
in hl« own terms and on his own volition the Information (variables) of 
importance. The advocates of this technique fear that a highly structured 
infonatlon collection format has the Inherent weaknesses of containing 
irrelevant questions and not Insuring the retrieval of all relevant Infor- 
mation. Still another "school" supports a combination or blending of these 
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two techniques* 

As to how these techniques should h© invoked, some feel that written 
questionnaires and check lists can be made to be self-explanatory or 
accompanied by written instructions and thus sent and retrieved through 
the mail without direct, personal contact. Others feel some degree of direct 
contact is required to retrieve information from the inquirer and, therefore, 
advocate the use of telephones. Still others fear that certain important 
elements of the educational environment surrounding the inquirer will not 
be recorded if field visits (on-site) are not made in the process of gather- 
ing information, and thus advocate this approach. Still others advocate 
combinations of the above. 

T. i alternative techniques to be considered and the alternative 
mechanisms for invoking them can be represented by the following matrix: 





Collection 
Techniques 
mentation^v^ (b) 
Me chani sms 
(Contact) (a) 


Highly Structured 
Check-Lists & 
Questionnaires 
(bi) 


Prose Statements 
(ba) 


Combination of 
Structured vso 
Prose Approach 
(bg) 


Written (Remote) 
(a) 


aibj 


aib2 


ai»3 


Telephone (Semi- 
Renx)te) 
( 02 ) 


“a*"! 


“24 


^2^3 


On-Site Visi J 
(Direct) 
U 3 ) 


ajbj 


CM 

X) 

CO 


®3’’3 


Written Telephone 
Combination 

(a*) 






^4^3 


Written u On-Site 
Combination 

(aj) 




ajbj 


ajbj 


Telephone On-Site 
Combination 

(ag) 






^6^3 


Written c< Telephone 
C: On-Site 
Combination 
(a^) 






a?b3 



Since PILOT UPIE is a$ r.iuch a research function as it is a ser/ice func 



tion, it is deslreable for all of the possible combinations to be tested 
eventually and evaluated on a comparative basis* However, practical limits' 
tions of time an<i r,ioney suggests that priorities be established as to which 
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combinations are tested first, and that certain combinations receive more 
initial emphasis than others. 

As to techniques for collecting information from evaluators of materials, 
it is generally agreed that written reports vtII^ be the information collection 
mechanism, and that structured check lists and questionnaires combined with 
prose state">ents be the collection technique. 

Limited Number of Instructional Materials 

It is generally agreed that PILOT fSPIE must begin with a limited num.ber 
of instructional materials rather than provide information on all instruction- 
al materials. This agreement has of late been made more specific by the 
selection of one curriculum product class — Elementary Science one inform 
nation transmission product class — overhead projectors — , and one infonna^ 
tion storape product class — overhead projectuals -- for inclusion in PILOT 

EPIS. 

The first of these, Elenientary Science, represents a curriculum area 
approach. That is, EPIE will develop the ability to simultaneously consider 
a multitude of product types within the curriculum area of Elementary Science. 
This will include both information storage and Infonnation transmission 
products. Vlith this approach, EPIE will be able to assist inquirers who are 
as yet undecided as to the particular product type they wish to use InanElomen 
tary Science course, or who are potentially interested in acquiring a mix of 
product types for inclusion in the curriculum. 

The decision to choose Elementary Science as the first curriculum 
area is based upon a number of factors. As pointeti out in an internal report 
on an lED-initiated survey of product information needs among a sample of 109 
school districts, 

MReadlng, although topping t' ) list in curriculum areas, was not 
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chosen due to its scope and complexities, the adviseability o^ 

T7aiting until EPIE can cover the field more completely, and 
[until EPIE c^iO incorporate • • • the results of current re- 
search projects* The decision to concentrate first efforts on 
Elementary Science was urged bv teacher associations and other 
professional groups, and justified by the questionnaire respon- 
ses placing this subject area second in importance [after read- 
in|J to schools *** 

Further, a majority of the consultants to EPIE have expressed a strong interest 
in the choice of Siementary Science, particularly from the viewpoints of need, 
manageability, and availability of information. 

The second and third product classes to be developed, overhead pro- 
jectors and overhead projector materials (projectuals), were selected for 

I 

reasons similar to the selection of Eleinentary Science* Particular reliance 
was placed on the recommendations of the Division of Audio-Visual Instruc- 
tion of HEA (DAVI)* Represptatives pf this group have already carried >ut 
extensive work in defining the important physical characteristics of these 

product types for inclusion in EPIE* 

Since the pilot system will be created within limited resources and 
time and with the goal of providing some rather than total service, it is 
likely that other mechanisms will be established to limit the number of 
products to be included initially within the three classes. Alternative 
operational strategies can be created and used to determine the specific 
prodqct make-up* Examples of factors upon which such strategies may be 

based are: 

a* Product type (within chosen product classes) 
b* Scope of application 
c* Innovative characteristics 

d* Availability (timing; for examination; for purchase) 
e* Awareness of availability (by EPIE) 
f. Product age 




g. Pedagogical Intent (explicit vs. implicit) 

h. Information system storage capacity 

i. Manufacturer size and mix 

Via analysis and consultation with schools^ the National Science Teachers 
Association, and the Division of Audio-Visual Instruction of the National 
Education Association, EPIE can continuously develop and apply specific 
product selection strategies as they become appropriate. 

Information Sources 

Information in PILOT EPIE should be a compendium of knowledge ob- 
tained from three major sources — producers, past users, and independent 
analysts or researchers. Inherent in this approach is the goal of providing 
the inquirer with information developed by people of different perspectives, 
capabilities, motivations, and experiences. Further, there exists the belief 
that EPIE must not appear to represent a biased source of information, which 
it would if it appeared to favor one or two of these three legitimate sources 
of information. In designing and operating the system, continual attention 
will have to be devoted to questions of specific sources of information with- 
in the general types and to the amount of information to be collected. 

The producer of each product will be expected to complete a profile 
of the product if he wishes to relate information other than that which can 
be objectively gathered and validated by EPIE staff from generally available 
data (e.g. - his evaluation of the teaching style nK)st appropriate for use 

of the material). 

The selection of past users for preparation of product evaluations 
becomes dependent uppn such factors as product type, quantity and nature of 
product evaluation and variance -explanation variables, degree and sophisti- 
cation of variance analysis, willingness of past users to provide evaluations. 
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2* For overhead projectors* acquire analyses of physical character* 
istics* to be perfomed by technicians in a ** laboratory** setting* 
in order to qualify for active use in information retrieval* Re- 
quire approval of the re search -technician reports by the Product 
Infoj:Tiation Advisory Corxnittees of KSTA and DAVI* Part of the 
analyses of physical characteristics my be in terms not related 
to application enviroa^iexits (CtS* - aptual luireiis), but should 
also be in terms relative to application GAviroir ’-euts Ce*g* - 
conditions under which actual lumens are effective, in terms pf 
room lighting, room slse, screen size, screen to projector dis- . 
tance, etc,)* 

3, For overhead projectual sets, acquire analyses of physical 
character! stjlcs unrelated to application environment* The analyses 
should be performed by technicians in a *' laboratory** setting* 

4, Establish in cooperation with the IJSTA and DAVI, policies and 
guidelines for the recruiti.ient and selection of researchers. 

Secure the assistance of the KSTA and DAVI in recruiting, screen- 
ing, and selecting researchers, 

Kature and Scope of Product Inforraation 

As to what information should be contained in EPIE with respect to 
the three product classes, agreement has not been reached on specific items 
of interest. However, there appears to be concern with the inform a tion con j» 
tent, appl i cat i on env i ronriient , and physical charac teristics, of Elementary 
Science materials; with the physical charact eristics and application enjdj^ 
ronment of overhead projectors; and with the physical characteristics of 
sets of projectuals (e,g, - a transparency series vs, an individual trans- 
parency in the series). For a definition of these terms and a description 
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and the depth of information retrieval custonlxation* As such, continual 
effort should be <|evoted to the develppment of past user selection criteria, 

in close cooperation with N3TA and DAVI* 

However, several guidelines are offered at this point which may help 
to expedite the realization of an operational system: 

!• In the case of all Eleirentary Science products and overhead pro- 
jectuals, include 

a* Evaluations by teachers who have used the product in a class- 
room setting 

b* Student^ who have used the product or been exposed to it in 
a formal course 

c* Curriculum design committees which have evaluated the product 
and selected it or recommended its selection* Evaluations of 
the product Itself by school principals and school system 
administrators v7ho have not been active in the selection of 
the product may not be meaningful. However, these people 
nay be valuable sources of information about the product 
users-evaluators \7ho fall under their supervision. 

2. In the case of overhead projectors, include evaluations by 

a* Audio-visual personnel who have selected, operated, and main- 
tained the projectors. 

b. Teachers who have used the projectors as a teaching aid in a 
classroom setting. 

In the selection of researchers to provide information, the following 
guidelines are offered: 

1. For Elementary Science products, acquire analyses of information 
content by logical analysts. 
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of a conceptual model for classifying variables which are thought to influenca 
product selections and evaluations ^ see Appendix A« 

In each of the above areas^ the consultants are working to define the 
variables of importance or inclusion* It is in general agreed that PILOT 
SpIE will initially permit a limited number of variables to be actively used 
in the performance of information synthesis, while others will play a more 
*'pissive” role In that they will be utilized for data collection and research^ 
but not for information synthesis* It is also believed that a system of 
^Sustoinized information dissemination should permit the user to declare those 
variables which are important, and to weigh variables himself in terms of 
their relative importance* Since, however, the information search and syn- 
thesis heuristics are not defined at this point in time, it is not yet clear 
that a strong need exists to actually utilize luiovzledge of user variable 
preferences and weights. 

Regardless of which specific variables are chosen for inclusion in 
the information system, each will be used in the context of one or more of 
the following categories: 

Producers 

- General Descriptive Information (Factual) 

- Recommended and IJot-Recor.nnended Usage Information (Evaluations) 

- History of Development 
Past Users 

- Recoiimended and Hot -Recommended Usage Information (Evaluations) 

- Description of Evaluator-Evaluation Environment Information 
Researchers 

- Recommended Usage Infonnation (Evaluation) 

- Testingj Infonnation (Physical Characteristics Evaluation) 

- Description of Evaluator-Evaluation Environment Information 
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No Product Samples 



It is senerally agreed that PILOT EPIE should not provide users with 
“hard copies” of products in which they are or may be interested# It is felt 
that many problems would arise if such service were attempted, not the least 
of which would be thQ higl^ cost to EPIE of procuring samples of the products 
included in the system. 



ADPITIOIIAL SPECIFICATION GUIDELI1>}ES FOR CUSTOIIIZED SERVICE 
Introduction 

As indicated earlier, one part of PILOT EPIE should be the development, 
testing, and evaluation of what has been termed '.'customized” information 
dissemination# To aid the s^em designers, some additional guidelines are 
offered in this section which relate specifically to this portion of PILOT 



EPIE, called "customized EPIE”. 

Limitation of System Usare 

Just as the information in the system initially should be limited to 



specific classes of r.atorials, so, boo, the use of the customized portion 



of EPI2 should be limited at the outset. Steps to define this limitation 
which have been taken so far are 1) the decision to restrict syster.i user? or 
inqv,irers to participating (paid) members who directly exercise or i.ifluence 



the decision-making process of evaluating and selecting materials, (excludes 
producers) and 2) the decision to restrict inquirers to individuals or groups 
meeting the above criterion whose school systems are within the geographic 
boundaries identified by those of uie ERIE and RB2 regional educational 
laboratories (Delaware, ,:iew Jersey, New York, Pennsylvania)# 

However, the planning of resources to service users cannot be effec- 
tively made without more explicit Imowledge as to the demand on the system# 
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Within the bounds of the user set currently defined* addit:iOhaX definition 
of de^nand will have to be perfon.ied which seeks to prQteot t;he economic 
solvency of EPIE and the desire of school systems within the four states to 
utilize customized EPIE. 

The selection of resources based upon demand forecastins is dependent 
upon the follovTing factors: 

1. Number of inquiries 

2. Frequency (spacing of inquiries 

3. Inquiry service ti:iie 

4. Inquiry response ti.;;e requirement 

5. Inquiry apeeptability incidence 

No steps have been taken as yet to forecast the values of the above demand 
factors arid their effects upon resource reqiiire.rents. A survey of potential 
syster: users may be helpful in developing sor.ie feelins hhe values of t^he 
above factors. However, such a survey would probably not be extremely help- 
ful because of the uniqueness of custoidzed EPIE and the low probability of 
potential users being able to predict their usage. 

The alternative is to defi^ie resourcevS by estimating what may 

be reasonable values for the demand factors. This process could become ex- 
tremely complex if in-depth consideration is to be devoted to predicting 

ft 

such values as the distribution of service times based upon analysis of in^^ 
quiry acceptability incidence; inquiry source, type, and product module ret 
lation distribution; neqd for repetitive dialogue, etc. In the belief that 
such in-depth analysis would be of little value and accuracy initially, it 
is suggested that the analysis be limited to producing the following opecifi' 
cations for computing initial (first year) resource requirements: 

1. Percentage of the school systems within the ERIE and RBS |?egional 
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laboratories which will submit inquiries. 

2. The average number of inquiries per school system submitting 

inquiries. 

3. The percentage of inquiries submit ted. which will fail to qualify 
for processing. 

4. The distribution of inquiries by month of receipt and month 
response due. 

5. The average time required to service an accepted inquiry, including 
time for information collection, inquiry processing, and report 
preparation (excluding general supervision). 

6. The average ticie jrequired to determine the unacceptability of an 
unacceptable inquiry. 

Once inquiry-serving resources arc acquired as a result of this 
analysis, they will represent limitations on the ability of PILOT c*PIE to 
service custoiized demand. For the nature of the resources (e.g. - personnel, 
equipment) will limit the ability of EPIE to respond in the short run to un- 
anticipated demand volumes. Mo such constraint will exist for the standard- 
ized information dissemination portion of PILOT EPIE, for varying demand 
(within reasonable limits) can be met by altering the number of copies print- 
ed of publications. 

Limitations on Product Information Utilization 

It is suggested that policies be created which initially limit cus- 
tomized EPIS»s ability and willingness to provide information on products 
within the selected areas. Several such policies which may be useful are 
presented below: 

1, Iiiniimjim Product Information Unit. 

First, the smallest product information unit within customized EPIE 
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should bs a ourchaseable Item. For example, If the materials within a series 
can only be purchased by buying the entire series, product information should 
be retrieved for the series as a whole rather than for selected components. 
This is not meant to imply that information abput various components of the 
series should not be available from producer's profiles, user reports, etc. 
Rather, the 'nfnrmation retrieval system should be geared towards the effi- 
cient retrieval of all information on a purchaseable product provided by a 
particular source. Further, the providers of information will be expected to 
relate their evaluations and reports to the total purchaseable item. 



2, IiltniriTUin Humber of Reports 

Second, customised EPIE should establish "lower thresholds" on the 
number of reports from a particular source type on a particular product which 
must be available before synthesised Information on the productv which draws 
from reports sub 3 aitted by the given source type, will be provided. For 
example, synthesised information which draws upon reports submitted by a 
particular type of past user of a particular product should not be used un- 
less a specified minimum number of reports on the product have been received 
from that type of past user. This policy is particularly necessary given 
customised EPIE's limited initial ability to carry out statistical analyses 
of comparative infowatlon in which the effects or limitations of saa,ple 

size would ’’automatically*' be accounted for* 

This policy will not restrict customised EPIE fro>„ providing infor- 
mation on products for which a sufficient number of reports exist from one 
or more source types. For example, if the lower threshold on reports from 
information analysis specialists (l.e. - logical analysts) has been met, 
but that for reports from a particular type of past user has not, 
matlon from the analysts would be used in processing the inquiry, but that 



from the past users would not. As a sufficient number of additional past 
user reports are accjulred to overcome the lower threshold > the total supply 
of past user reports would be »»activated‘» for inquiry processing. In this 
way, the inquirer will tre protected from receiving synthesized information 
which is unsupported by a reasonable amount of detailed information. At the 
same time, customized EPIE will not be restricted from considering a product 
when processing inquiries until sufficient information on the product is 
available from all three of the major source types, 

3, Minimum Potential Expenditure, 

Third, customized EPIE should seek to achieve economical utilization 
of its limited initial resources by requiring a minimum potential expenditure 
to be associated with an inquiry before the inquiry will %e processed. This 
caii be achieved by establishing a formula which takes into account the unit 
cost of the product (or a "typical" unit cost in the case of multiple items 
within a product type), the quantity being considered for purchase, and the 
source of the inquiry (e,g, - a county school district vs, an individual 
school). The letter factor is intended to represent the probable extent to 
which the purchase would utilize the total educational products budget of the 

education unit represented by the inquirer. 

The formula should be created with the goal of avoiding the use of 
SPIE’s initial resources in processing inquiries which represent a very small 
percentage of the products budget under the control or influence of the in- 
quirer. Thus, whereas a teacher may be permitted to submit an inquiry re- 
lating to a potential expenditure of $25, a superintendent of a county school 

district may have the same inquiry rejected. 

This policy is analogous to advising a large business not to hire a 

management consultant in order to save $1000 by solving an operating 



problem. However, It also recognizes that there are different sizes of 
businesses, and that saving $1000 may be extremely worthwhile for some of 
Che smaller ones. Since there will always be a cost associated with the use 
of customized EPIE's time and resources, and since the time and resources 
required to process an Inquiry will not be directly proportional to the 
amount of potential product expenditure underlying the Inquiry, this policy 
Is believed necessary for the nalntalnance of a reasonable charging structure. 

4. General In formation Contenc Retrieval 

Fourth, customized EPIE should Initially operate under a policy of 
general rather than highly specific retrieval on the subject matter portion 
of the Inforniatlon content of a product, For example, a general science text 
may contain a chapter which discusses the science of meteorology. Vlthln 
the chapter, alternative methods of forecasting the weather may be presented, 
with one such method being probabilistic forecasting. Customized EPIE should 
not Index initially the contents at the "alternative forecasting i.«thods" • 
level, but rather at the "i,mteorology" level. As In an earlier example, this 
doesn't mean that one or more reports from past users of the text nay not 
have cosmmnted on or .. -aluated the treatnmnt of probabilistic weather fore- 
casting. Hhat Is implied Is that the normal search and retrieval methods of 
customized EPIE will not permit selective searching for solenca texts dealing 

with probabilistic weather forecasting* 

Ihe normal procedures may lead to the retrieval of the names of a 

number of texts which discuss the science of meteorology. A policy should 
be established to determine whether additional EPIE staff time should be ex- 
pended to "hand search" (l.e. - scan) the selected texts to determine If one 
or more discuss probabilistic weather forecasting, or whether this task should 
be left to the Inquirer. Since past use.- reports will be available on the 
selected texts, the possibility still exists that a review of these reports 
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will produce information on a particular text's treatment of probabilistic 
weather forecasting, If the evaluator chose to highlight that portion of the 
content. At this point, however, such detailed information retrieval becomes 

coincidental rather than planned. 

If additional BPIE sta^f time is expended on "hand searching^*, a 
policy should also be available for determining whethe «»ddltional user charges 
are to be assessed, either in the form of monetary charges or time charges to 
be applied against the users "account balance" pf available time. 

5, Product Selection vs . Non-Product Selection Information 

It is suggested that customized EPIE should organize its Information 
into tx 7 o major retrieval categories: product selection Information and non- 

product selection information. Product selection inforr.iation is that which 
should be available for use in deriving a selected list pf products to be re- 
ported on in response to inquiries. Once a selected number of products have 
been chosen for reporting, the non-product selection Information on those 
products should be used for abstracting or direct reporting to the inquirer. 

Often, the non-product selection information will repi^esent a deeper 
dimension of a particular class of product selection information. For example 
whereas the product selection Inforaation may Include author's name, the 
non-product selection Information may include information on the author's 

background, his affiliations, etc. 

Non-product selection infonnatlon should also include information . 

which the designers of EPIE feel is not likely to be required or logically 
used to develop a list of selected products most likely to be of interest 
and value to inquirers. For example, whether or not the author of a text 
received f inant lal support for writing the text may not be expected to be 
important in developing a selected list of products. 



The distinction is important j in that different inforiiiation storage 
and retrieval techniques (e.g. - equipi.ient, coding, storage media) ^^ill 
probably be required ^or product selection and non-product selection infor- 
mation* 

nature and Scone of Infornation Sy Inquirers 

The lnforr.iation to be retrieved from inquirers should conform to that 
provided by evaluators. Just as producers should provide information about 
the cost of products, the inquirer should be expected to state how much he is 
willing to spend, if he has a limited budget and it would bo of little value 
to report in detail on products he cannot afford. Likewise, the success of 
the product selection process will be dependent upon his ability to state: his 
requirements, preferences, and application conditions in terms conforrilng to 
those In Tihiebt products have been evaluated. The extent to which he does this 
vTill influence SPIS»s ability to provide synthesised information on products 
likely to bo found of value. 

It is likely that all inquirers will not respond initially with equal 
scope and attention to SPIS^s requests for information. Tnus, it will be 
necessary to have policies to determine if sufficient information has been 
obtained from inquirers to provide a meaningful service. Sonre guidelines for 
initial formulation of such policies are offered below. 

1. If a “pass” of the product selection information using the infor- 
mation provided by the inquirer fails to vield a minimum propor- 
tion of the total products potentially reportable, require the in- 
quirer to ass-gn priorities on the inforaatlon he has provided. 
Beglrning with the lowest priority information, remove information 
from considera1;ion in additional passes until the above require- 
ment is i;. 3 t. Once the requirement Is met, determine how the 



**vaXues*' of the deleted information for the selected products 
differs from the "values” initially specified and repo^rt the 
diffv^rences to the inquirer. The total products potentially 
reportable is the sum of the products in the EPIE system within 
the product type if specified by the user, or the sum of all 
products within EPIE and the appropriate product module if un- 
specified. 

2. If a pass of the product selection infom^ation using the informa- 
tion provided by the inquirer fails to yield a maximum proportion 
of the total products potentially reportable, require thp inquirer 
to submit additional information if he has not completely re- 
sponded to SPJE*s request for information, or if he has resf nded 
in terms uninterpretable by EPIE. Invoke this requirement until 
such time as the above requirement is met, or until EPIE feels all 
information has been provided, or until the inquirer declares him- 
self unable or unwilling to provide additional information. If all 
information has been provided or if the above reqqirement has been 
jtiet, proceed nor.iially. Othen^ise, response to the inquiry should 
be limited to reporting the descriptive (factual) information on 
the products selected with the available information, with no 
evaluative information to be provided. 

Flexible Information Retrieval 

It is suggested that customized EPIE should be designed to create a 
highly flexible infon.\ation retrieval system. The system should be flexible 
in the sense that it should be capable of efficiently handling via normal 
retrieval procedures diverse inquiries of high frequency. 

For example, customized EPIE anticipaces receiving frequent inquiries 
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ou the use of Elementary Science Hts, without r$fere?ice to specific ones. 

It should, therefore, have a retrieval system which V7lll pcnnlt the cpnsid* 
eratlon of all kits, in the sense that the search and selection process should 
begin with the **universe'* of infon.iation on available kits within the EJPIE 
system. Conversely, EPIE also anticipates the receipt of inquiries relating 
to a specific type of Ult (e.g. - a botany kit), or pven a particular manur 

facturer’s kit. As such. It should also bq able to efficiently puU from the 

I 

Infonnation file only that information which pertains to the particular "sub- 
set” of the kit "universe" of interest. 

Further; just qs customised EPIE should be able co Vslice the infor- 
mation plo" by product, soj, too. It should be able to do so by curriculum 
and by information source (i.e. - producers, ps^t use:|ps, researchers). To 
achieve this flexibility regardless of the use Qf automated equipment — 
will require the careful design of indexing procedures and coding schenes. 
Design guidelines for achieving sueh flexibility will be offered in a later 
section. 

Adaptive Infoniation Retrieval 

In addlt.v>n to designing an information retrieval system which is 
flexible in the se.ise described above, the system should alpo be adaptive 
In two ways. 

First, recalling the earlier discussion of p)?oduct selection vs. non- 
product selection information, an efficient system wJH psjobahly employ 
different techniques for classifying and hanuUnc tnese tvq type© of infor- 
mation. Information considered as non-prqduct selection will probably ho 
encoded In limited ways, whsreap product selection infop-iation will probably 
be highly encoded* Also, normal search procedures of product selection 
Informat i 0.1 may entail some use of electronic data handling equipment, such 
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as sorters, collators, or computers, and manually operated coordinate Index- 
ing systems (e*a* - Peeic-A-Boo system), whereas non-product selection infor- 
mation search procedures i.iay be constrained by standard classification index- 
ing systems designed to organize written material stored in filing cabinets. 

The design, however, must be prepared to respond to experience in using 
the system. This may lead to the recognition that information which was 
initially designated for use in product selection is best reclassified as 
non-product selection, or vice versa. As this occurs, mechanisms must exist 
for altering the way in which a particular set of information is treated with 
respect to coding, location, and applicable search procedures. 

Second, the system must also be adaptive in the sense that information 
can be added to or deleted from the system without necessitating the revision 
of indexing procedures, the redesign of coding formats, the refiling of infor- 
mation, etc. In those instances where standard classification systems are 
used such as sequential coding (i.e© - information coded and then filed in 
numeric or alphabetic order), the adaptability design criteria vrill require 
projections to be made as to the aniouni; of space, nuiaber of colur.ins, number 
of miassigned positions, etc. which must be planned for in order to permit 

grow’tli without redesign or reassembly. 

Alternatively, coordinate indexing systeiAS usually perr.iit a greater 
decree of growth allowance v'itliout redesign than do standard classification 
systems, and with less need for attention to planning growth allovrances. 

Usually inlicrcnt in one carci of a Peek-A-Boo syste*.i are ;.iany mused or un- 
assigned locations which represent a growth allov’ance. Further, as one desires 
to expand the depth of indexing information (o.g. - incorporating into the 
retrieval system the ^’probabilistic weather forecasting” level), this can be 
easily done in a coordinate indexing system by adding more Poek-A-Boo cards 
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to tho esistlnc; deck, whereas such expansion under a standard classification 
systo.i :..ay require the redesign and refabrication of a substantial portion of 
the infer.., atlon retrieval syste.„. For exauple, if the standard classification 
system is superir, .posed upon a punched card operation, all cards will have to 
be repunched if sufficient unused coluims do not exist to penr.it the length- 

enirv^ of a code nu^iiber* 

conversely, if it beco.es desireable to delete certain Information 
from the system, a standard classification system may require revision if 
failure to do so leads to excessive unused positions, colums, file space, 
etc., or excessive manipulation of ”dead« information. A coordinate indexing 
system can be designed so that the procedures for using it automatically ex- 
elude from manipulation or consideration "dead" information. SU«e, however, 
advantages of higher processing speed and error cheeking raay be Inherent in 
certain standard classification systems, the initial design may be a blend of 
the two main fonus of information organisation to achieve the advantages of 

each* 

Product Selection Infonia t ion Svstei., Peslgn Criteria 

The above section discusses alternative information organisation 
schemes for purposes of storage and retrieval of product selection informa- 
tion. In particular, standard classification systems ,nd coordinate index- 
ing systems are briefly discussed as to their relative advantages and dis- 

advantages* 

At this point in tine, two -approaches to the design of an initial 
product selection lnfor..,ation syste-,,-. are considered feasible in light of po- 
tential demand and resources. It is suggested that these two alternatives 
be explored In depth by EPIE's information retrieval system design group. 

One is a £U^ coordinate indexing system operated manually, the other is a 
combined standard classifioation-coordiM.red Indexing f-.tem operated 
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mechanically thtough the use of punched card equipment* As the major guiding 
principle for the eicploration pf these two alternatives, it Is recommended 
that both systems be designed such that they possess equal flexibility and 
adaptability* Specifically, flexibility and adaptability should be equal in 

tcrr.iG of ability to: 

1* Add or delete products from the systpm» 

2* Carry out product selection search and retrieval on equally de* 
tailed levels describinc the source of evaluations* 

3, Carry out product selection search and retrieval at the evaluator 

descriptive level specified by the Inquirer or essential to 
achieving a list of selected products according to the policies 

recommended* 

4* Carry out product selection search and retrieval on equally de- 
tailed levels of evalUvation* 

Although a coordinate indexing system is inherently more adaptable than 
the standard classification portion of a punched card standard classification- 
coordinate indexing system when it cotiies to expanding the number of categories 
within a particular information type and level, the latter should be consid- 
ered sufficiently adaptable if it is apparent that tl)e number of categories 
which can be achieved surpasses any reasonable estimate of the maximum number 
liUely to be required* For example, within the information type and level 
of ’’teacher subject matter competence” (e*g* - ’’products recomcAended by pro- 
ducers for teachers of ’average* subject matter competence”, or ’’teachers of 
’average’ subject matter competence recommeiK-d these products for use with 
students whose attitude towards the subject saatter is ’mildly interested’”), 
if five categories of ’’teacher subject matter competence” are established 
initially and two columns are designated on a punched card for recording 




"teacher subject matter competence" in numerically coded form, allowance 
exists for recordlnc up to 99 different categories. Even though the coordi- 
nate indexing approach is isiore adaptable in that !»K»re than 99 categories of 
"teacher subject natter" competence can be incorporated without redesign or 
alteration of other ?eelc-A-:ioo cards, while the designation of the 100th 
category would require a third colum on the punched card which may already 
bo in use, the latter should be considered equally adaptable in practical 



terms. 



In exploring the two alternative approaches, the relative advantages 



and disadvantages of each should be considered as follows. 



given the above de- 



sign criteria which "normalize" other iiiherenu 



advantages or disadvantages: 



System Tvoe Advanta.q;es 



rure Coord. Index, !• 
Manually Operated 

2 . 

3. 



Loss need to predict riax- 
inuu number of categories 
within an information type 
and level. 

Search procedures very 
simple. 

I^ss costly and time con- 
suming to design. 

Less capital expenditure. 
2oquires less shilled sys- 
tem operators. 



Disadvanta::es 

1. Search and selection rel- 
atively slow. 

2. Frobability of selection 
errors higher. 

3. Potentially more direct 
labor expenditure. 



Combi'aod Coord. 

I icTox. Standard 
Class i f i cat i on , 
lachiue Operated 



lo Potentially faster search 
and selection. 

2. Probability of selection 
errors lower* 

3. Potentially less direct 
labor. 



1, Potentially excessive re- 
dmidancy in card handling 
(i.G. - sorting, stacking, 
collating, refiling, etc.) 

2, - Greater capital exfienditure 

3, Seed for higher skilled 
cyste:.! operators. 

4, Sore costly and time- 
consumiiig to design. 

5, Search and selection pro- 
cedures more complex. ' 



The major challenge in designing a i.iechanized combined system is felt 
to bo the design of efficient search, selection, and refiling procedures which 
will avoid excessive card handling. Too many punched card systems which are 
ostensibly more efficient than manual systems lose whatever advantage they 
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r*ay have by incorijoratina poorly designed search and selection procedures. 

As to achieving acceptable dosi/^n flexibility and adaptability, initial 
efforts indicate this can be acconplished* i'cr a description of a punched 
card conbined coordinate indexing - standard classification product selection 
iiifori'iation systor.i, soe Appendix -• 

3^nithesis Procedure Criteri_a 

As Mentioned earlier, once the product selection inforr.iation system has 
produced a selected number of products for inclusion in an inquiry response, 
the non-prodvct selection information system should be activated to prepare 
the actual response. To assist in the creation of detailed design specifica- 
tions for this portion of customised EPIS, the following guidelines are 

offered: 

1, Reports should infor.i the inquirer of the extent to which the pro- 
ducts selected conform to the analysis specifications vTiiiich he pro- 
vided, 

2, Products selected on the basis of past user evaluations should be 
supported by information describing the past users (e,g, - descrip- 
tion, lurmber of reports on file, etc,) 

3, Reports should inform the inquirer of the values of those ”reco:n- 
i.endod usage'* variables not specified by the inquirer for use in 
product selection, 

4, The system should incorporate efficient i>rocedures for searching 
and updating information. This may require the use of punched card 
equip:.aent for periodic updating of suifLiiary data, 

Tnformation Flow Channels 

In previous sections, sources of information and types of reports for 
customised EPI3 have been discussed in various contexts and examples. At this 




poinv, It ifl desireable to state In a conpact and simple manner the 
probable sources and receivers of InfonnaClon, plus how the infonnatlon 
will flow. The relationships between infor/jation sources and flow are 
rooresontod in the followinc flow diaarar. of Inputs and outputs: 
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SIMPLIFIED FLOW DIAGRAM ■■ CUSTCMIZED EPIE 




2 - Product Specifications 

3 • Analysis pf Information (Logical Analysis) 

4 «• Analysis of Physical Specifications (Lab Technicians) 

5 • Product Xnj^ormation Inquiry 

6 •• Response to Produqt Information Inquiry 

7 - Product Applicat^ion Evalpation (Performance) 

8 - Product Application Evaluation (Synopsis) 
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The total Liforjnation flow between r.ieinbers of the system is not 
represented i.i thij? dla23*nii, as certain infon^ial direct comr.'.unlcation lines 
will probably e::ist (e.ii. - industry^ schools). Rather, the diagran repre- 
sents the i.iajor or forwal coifiiaunication links \7l1ich are probably essential 

to the perfornanpe of ^IPIE*S functions. 

The diai^jran also does not convey all feedback and self-evaluation 
i)rocQ?SQ^ \ 7 hich \7ill hopefully ensue. In particular, ERIE will seek feed- 
back froi.1 users of the system as to their satisfaction with its services, 
the entent to which it has contributed to improved product selection, and 
its needs for j.iodlf ication and ii.iproveuent. 

Finally, the diagram does not convoy tfie process of Iterative dia- 
logue which is likely to ensue between custOxaisod SPIE and its users in the 
processing of ixiquiries. As customized EPIE finds txiat initial dialogues 
do not produce sufficient information to permit useful synthesis of the 
product infornation it maintains, or as inquirers find tnat initial re- 
sponses to inquiries do not provide sufficiently precise and useful iiiforna' 
tion to liiakn decision^, additional dialogue will nopefully ensue. Particu- 
larly at t'vy outset of custo.’iiized EPIS, stich iterative dialogue should be 
doiiinant. Over time, hov 7 over, as customized EPIE ii.iproves its knowledge of 



information which it 



iiiust obtain froiu inquirers to perform meaningful re- 



I trieval and jynthesis, its techriques for eoliciting such inforr.iation, its 



statistical analysi 
users learn !iow to 
uniform techniques, 



s capabilities, and its information base, aiid as systo .1 
precisely state inquiries and collect information using 
the need for such repetitive dialogue should diminish* 



GROWTH CHARACTERISTICS OF PILOT EPIS 
Gince PILOT EPIE is intended as an initial operating system which will 
continuously r.jove closer t® achieving its long-range objectives, its designers 
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rftust be concerned w.',th defining specific ways in which it can and should 



grovT. 



Contiguous Evolution 

The most important st°wth policy to be stated is the concept of 
continuous e^>^olution » The goal of the system designers should not be 
the creation of a pilot syetevA wl w|U operate intact} for a pre-specif ied 
time period, finally to be replaced by a ”new mode?.#" Rather, it should be 
to create a flexible, adaptable system which possesses sufficient resources 
to generate improvement and expansion on a continual basis. In an earlier 
section, the criteria of flexibility and adaptability as they apply to PILOT 
5PIE were more fully explore^!. 

To some extent, however, time is bound to become a dominant factor in 
regulating the evolution of EFIS. To maintain the interest of schools and 
industry, EPIS must be cojmnitted to evidencing growth at least once yearly. 

In the fall of each year, as schools begin to consider educational product 
needs for the following academic year, hopefully, those who have used EPIE 
the previous year can continue to do so in new curriculum-product areas. 

Those who have not been able to use SPIE in previous years will also, 
hopefully, seek service in current curriculum-product areas. 

Further, the process of feedback from schools and that of self-evaluation 
which EFIS plans to carry out, although continuous, will in part be time- 
regulated. There is an inherent tivne lag between an EPIE-influenced 
decision and an evaluation of the decision, its effect upon the educational 
process, and the product chosen. Typically, a decision reached in the winter 
of one year to Iraplement a product does not come "full cycle" until the 
summer of the following year, creating a time lag of approxiraately fifteen 
months. Product application evaluations frora users of EPIE cannot^ therefore, 
be used to update the information files and retrieval procedures until fifteen 
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months subsequent to a product selection decision* EPIE will become 

at least partially time-dependent by necessity as well as by design. 

Modular Growth 

VJithin the growth concept of continuous evolution is that of modu3ar 
growth * For althou^ EPIE n)ust evolve continuously, it must also seek some 
structure to its evolution* The concept of piodularity is that EPIE should 
expand by defining new major classes of products along product type and 
curriculum lines as growth units* Just as EPIE has decided to begin with 
the modules of Elementary Science and overhead projectprs-projectuals. in 
the future it must select for development new product modules such as Social 
Science and closed circuit television-video tapes* 

The reasons for this approach are two-fold* First, from the viewpoint 
of EPIE and its developers, any approach other than the selection of an 
educational product area in its entirety would be both chaotic to manage 
and of disservice to system users* Second, from the viewpoint of thp users 
of EPIE and its information sources, an approach which dealt with only a 
segment of a product area could create an image of EPIE being a pystem 
with inherent bias and thus threaten its credibility and survival. 

Professional Association Relationships 

Thus, EPIE seeks to develop information "packages*" It is doing so 
by establishing formal relationships with professional educational assoc- 
iations which are particularly concerned with a product module and willing 
to provide guidance in its development. The relationships it has established 
with the national Science Teachers Association (NSTA) for the Elementary 
Science module and with the Division of Audio-Visual Instruction (DAVI) of 
the National Education Association (NEA) for the overhead projectors-project- 
uals module are considered instrumental in the full, objective, and competent 
development of the mod‘;«.Ies and in the preservation of EPIE*s image as an 
organization dedicated to the fair representation of all interests# As EPIE 



seeks to develop new modules in the future, it should concurrently seek the 
establishment of formal relationships with appropriate professional associations* 

Qualitative Growth 

It is hoped that EPIE will grow along qualitative as well as quantitative 
lines* Particularly within the areas of automation, depth of infosmiation 
indexing, statistical analysis, and information retrieval techiiiques, EPIE 
should continuously seek to improve itself and its service within existing 
product modules* As stated earlier, the success of EPIE will also be dependent 
upon its ability to develop meaningful, acceptable, and consistent instruments 
for evaluating products and their performance* 

User Boundary Expansion 

Finally, EPIE should expand the boundaries of its user 'universe-* by 
increasing its resources such that school users within the current geographic 
boundaries and without can utilize customized EPIE either for the first time 
or wre frequently. EPIE also should seek the day when it has sufficient 
resources to permit individuals or groups other than schoql personnel, such 
as -iublishers and manufacturers, to su.bhdt inquiries* 
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Since the sajor purpose of this paper is the presentation of guidelines 
for developing design and performance specifications for PILOT EPIE, intensive 
effort has not been devoted to an in-depth analysis of inplementation require- 
ments* Further, cost estimates for developing, implementing, and operating 
PILOT SPJS are not offered here, although assistance has been provided with 
the developitent of a budget for PILOT EPIE, as presented in the proposal to 
which this paper is an addendum. However, the major implementation activities 
to be performed in the development of customized EPIE are listed for purposes 
of documenting the areas which will probably require further exploration and 



planning* 
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The activities to be 9ar3;ie(J out are suggested as .follows: 

1* Acquire the product information VThich V7ill bo stored in customized EPIS 
(also ..infomation base for standardized EPIS)* This will require the 
selection of actual products for inclusion in PILOT EPIS; the preparation 
of evaluation-perforriance reports by producers, past users, and researchers; 
ai)d the polleption of infonnatio^i by f|PI2; staff* ^Iqo, pre-requisite to 
the dcvelopnent of product reports is the do ter.iu nation of specific infor- 
mation requiremonts, value ccalos, ovaluation-ineasureipeiit -lethodo logics, 
and data cqllection i.nstru''ents* 



2* Deteri'ine tho apprepri a tone cs of the guidelines for developin.; sign and 
parfor;.;ancc specifications sei; forth in this paper and modify as necessary* 

On the basis of the final guidelines, develop operating policies as indicated* 
3* 'Uthin the approved guidelines and operating policies, develop the infor- 
nation storage and retrieval syste :s to a level sufficient to achieve 
operational status. T'lc n?.jor areas of activity for t!ie infonnatioii 
storage and retrieval design group are thougiit to be as follo*’r.! 
a* Indo;:ing-f i 1 ii ig let' lOds . 
b. Infor.;ation organiza io:i. 



Cod in 



pro CO 



■•ure 
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d. lHfo::..;ation handling procedures, 
o* gy.VdiGsis decision rules. 



f. For )S design. 



g. 3o,.tiOls design. 



h* Training for system operation, 
ie Demand forecasting. 

4. ’'ased ipon the outcomes of the detailed infoi* .ation storage and retrieval 

dcci^ ;, dctor..tine requirer.or.tc for housing, equip;. out, supplies, and ^:ersonnel* 
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Acquire and train iiecosaary personnel* Principally, these would be people 
eUiiaced in the preparation of i^rodnct evaluations (e* 3 * - logical analysts, 
student interviewers) and in the operation of the custonized infomation 



s7ste.;i. 



C, 



Conduct additional proriotion of potential system users via progress reports 
published in staudardised 3FIS and via seninars. Develop and engace in an 
educational-trainii\, progra::. on the value of customized EPIL and how it 
is used# 

7# Carry out a trial run of the initial operating systorn for customized EPIE 
for pv poses of testing and refine lent# 

As to iviplonentatioii responsibilities beyond general administration, it 

* 

is suggested that full-tine systems enjineers be acqair^-1 and designated as 
the Infor;..ation Storage and Pvetrieval Systems Design Group, with responsibility 
for perfor..:ing the detailed design V'orh as set forth in step ir3 above. In 
addition, this ^roup should have responsibility for the analytical activities 
to be performed under stops ,r2, A, and 7 above. Analytical responsibility 
for steps H, 5, and T. should rest with other staff, to be designated the 
luforuation Acquisition Group, The two staff groups should work in close 
cooperation, as their activities are inter-related and inter-dependent. In a 
very real sense, the success of PILOT EPIE will hinge on the ability of its 



aduiuistratox's to effectively coordinate the work of these two key staff groups. 
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APPENDIXA 



Product Svaluation Conceptual Model 

A major Issue being faced at this stage In the devei^oj^ment of EPIE Is the 
selection of variables for Inclusion and operation within custcmlxod EPIE, This 
process has been assisted to date by bhe development of a conceptual model for 
clarifying variables which are thought to influence product evaluations. In 
the belief that this conceptual mpdel will contlnuQ to sep^e as a device fv/'' 
logically grouping variables, its basic elements are herein described. 

The conceptual model is built upon the assumption that evaluatorr of a 
product will differ in their evaluations and that these differenpes pan be ex- 
plained or accounted for by Identifying va-ious caus^ of the differences and 
the AffArhs of each cause . If this is true, EPIE as an information system will 
gain perhaps the most important capabll' y of all — the ability to predic t how 
a given product will be perceived to fair in a given "situation*'. For by 
Identifying the *' situation*' surrounding a particular inquiry In terms cor.^- 
patable with the definition of "situations*' surrounding existing or historical 
product applications, EPIE will be able to "match" inquiries to products so as 
CO identify either those products likely to gain acceptance and success in the 
Inquiry "situation** or those products which may not and the reasons why. The 
extent? to which EPIE can function in this manner will directly influence its 
effectiveness and image as ai^ objective source of informatj^, acting in a 

"staff" capacity to the "line" decision maker. 

The relationship envisioned is not unlike that of the guidance counsel- 
lor to the student. Just as the effective guidance counsellor enhances the 
quality of the students decisions on future academic ar\d professional endeavors 
by clarifying alternatives, providing information, and by assisting the student 
in learning how to evaluate potential courses of action, so, too, EPIE in its 

most effective form will $erve as a source of pragmatic Information displayed 

A 
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in forms amenable to administrative evaluation, and as a "guidance counsellor" 

a 

in improving decision*making skills* 

Four basic classes of variables have been chosen to represent the "sit- 
uation" or causes of differences in evaluation* These are physical character- 
istic variables, information content variables, environmental application or 
interaction variables, and evaluator variables* 

Physical characteristic variables are those variables which describe the 
physical characteristics of the product — its weight, dimensions, type of print, 
use of colors, tensile strength, etc*, depending upon the product* Such variables 
can account for different product evaluations* For example, if 100 past users 
of an overhead projector applied the projector in identical situations, with the 
one exception of 70 users being right handed and 30 being left handed, and the 
70 right handed users all evaluated the projector as "excellent" whlls the 30 
left handed users all evaluated the projector as "fair" (using the same evalua- 
tion procedure), one may suspect that a physical characteristic accounted for the 
different evaluations* If upon examination the projector was found to be con- 
structed with the switch panel on the right side, one may intuitively conclude 
that the location of the switch panel was thfj "situation" variable accounting 
for the different evaluations* Further, using statistical techniques, one could 
both quantitatively represent the relationship between this variable and the 
evaluation and forecast the probability of a future left handed user, operating 
in a "situation" identical in every respect to the "sample" left handed users, 
finding the projector "fair" by the same evaluation procedure* 

In the case of information tranamission products, such as projectors, 
blackboards, television sets, audio sets, etc*, physical characteristics 
probably become a dominant factor in evaluations* However, in the case of 

products which are primarily of an information storage nature, such as maps, 
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textbooks, audio tapes, video tapes, transparencies, slides, etc, one vould 
suggest that the information content of the product Itself would strongly in- 
fluence the evaluation. Thus, a second major class of v'arlablesexlsts to de- 
scribe the situation - the Information content variables. These variables 
would describe what Information Is contained In a particular product, both ex- 
plicit and Implicit. That Is, a product may contain an historical account of 
the Civil War, but "beneath the surface" may also be the author's expression 
of his views on the Injustices done to Negroes in the United States and on the 
«,sponslbllltles of a Federal Gcem^nt In matters of civil rights. In this 
case, a set of evaluators may arrive at differences of opinion about the pro- 
d...ct based upon their satlsfacf.cn with the Informati o n presented. 

This class of variables would Include those which describe how Informa- 
tion is presented as well as what Is presented. Many educators feel the method 
and quality of presentation can Influence a product's acceptability as much as 
what it says. For example, an evaluation may show that highly accurate peda- 
gogical information has been presented In a style of writing which Is awkward, 
that the author's vocabulary exceeds the level which student users of the mater 
lal can be expected to posses-, and that the grammatical structure Is filled 
with errors. Such an analysis would undoubtedly Influence an evaluation of 
the material. Following the concept of explaining differences In evaluation, 
another evaluation by an evaluator Identical In every measureable respect to 
the first may find the style, vocabulary, and grammatical structure of the 
material perfectly acceptable, and as such differ In hls, evaluation from that 
of the first evaluator. In this case, one would conclude that the "how pre- 
sented" Information variables explained the differences In evaluation. 

It must also be. remembered that all Information storage products are 
themselves physical entitles and as such have physlpal characteristics as do 
information transmission products. Thus, causes of differences In evaluation 
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nay lay in the physical characteristics of a prodtxt a£ we?l as the information 
characteristics^ '’o dearer example could exist to demonstrate the continuing 
importance of physical characteristics In evaluating products which are primarily 
of the inforraation storage type than that of the highly educational film which 
continuously breaks when in use because inferior materials were used in Its man- 
ufacture* 

Yet these variables alone may not account for all differences in product 
evaluations. Another factor thought to be of major contributory importance In 
differing product evaluations is the evaluator himself. As EPIE anticipates the 
evaluation of individual products by more than one person, it must be aware of 
the "bias” or differences attributable to the evaluators themselves. Major dif- 
ferences in motivation, experience, intellectual levels, and analytical skills 
are expected to exist between the three major groups of evaluators — producers, 
product users, and researchers or analytic specialists. Even within one of 
these major groups further differences are also anticipated, particularly in 
“le product user class. Comprised of teachers of different educational and 
sociological backgrounds, principals, purchasing agents, superintendents, cur- 
riculum specialists, and students, this group is expected to produce evalua- 
tions which differ at least in part because of evaluator (as opposed to mater- 
ial) characteristics. Thus, any information system which attempts to explain 
differences in evaluation must anticipate the possibility of evaluator effect 
or "noise". 

In some instances, the evaluator may simultaneously be a direct user of 
the material in an instructional setting, as in the case of a teacher or a 
student. When this occurs, it is felt by leading educators that characteristics 
of the teachers and students themselves will effect not only the evaluation of 
products, but the academic performance of students under their usage. Thup, 
intertwined in the evaluation process and effecting its outcomes, are teacher 
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and student characteristics, either because they perform the evaluation, or be- 
cause they are a part of the setting in which the product being evaluated was 
used. In the latter case, the effects they impose upon differences in evalua- 
tions are classed as environmental application or interaction variables. 

Specifically, such factors as the teacher knowledge of subject, his self- 
pedagogical image, his methodology of teaching, his experience, the student’s 
sociological-cultural-economic background, the student’s apptitude and e:?per« 
iences, etc, are thought to be variables of significance in terms of the per- 
formance and evaluation of educational materials. In addition, other variables 
for possible inclusion in the environmental application or interaction class are 
those which depict the environment surrounding the teacher and student, spch as 
community structure and policy, physical plant, the educational institutions 
attitudes towards and experience with "innovative” instructional materials (e.g, 
kits, television), and the classroom size. 

Within this class of variables, the particular ones of importance and the 
"scope" implicit in them will depend upon the products being considered and the 
questions being asked. For example, an inquiry directed towards the acquisition 
of instructional material to supplement a curriculum in Svex education would 
probably generate the consideration of "community attitude towards sex education" 
as an interaction variable of importance. On the other hand, an inquiry direct- 
ed towards the acquisition of overhead projectors for multi -curriculum applica- 
tion may require the inquirer to provide 2PIE with information on the location 
of projectors in relation to the projection screen (distance), the size of the 
screen, and the room lighting conditions, in order to search for projectors 
which are intended for use under the stated interaction conditions or which h^ve 
been used under like interaction conditions. Thus, environmental application 
or interaction variables are those variables whi«"a in some majmer correlate 

the educational product with the educational environment in which it has been, 
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may or is intended to be applied. 

It is also important to note that in certain instances, the concam may 
be with the interaction of an information storage product with an information 
transmission product, as in the case of movie films and projectors. As th|a 
interaction takes place within the application environment, the variables which 
describe the interaction (e.g. - film sprocket holes to projector sprocket fit) 
should be considered as environment application or interaction variables. 

The diagram below is intended to represent the model concepts discussed 



above ; 



PRODUCT SVALUATION CONCEPTUAL MODEL 
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Partial Description of a Punched Card Combined Coordinate Index ■ Stand ard ClasA^ 

Product Information Selection System 

The basic information storage unit of the system should be a 12 row, 

80 column punched card (e,g* - IBM card) which can be processed on sorters, 
collators, key punches, tabulators, etc. A specific number of beginning 
columns should be reserved for the entry of standard classification system 
code numbers or alpha«numeric entries (e «g* 7A328, AR293) which identify 

each card. In many instances, each card will represent those products which 
have been rated for a particular position on a value scale for a particular 
category of product selection information, according to a particular evaluator 
type. For example. If a nine-position scale were designed for recommending the 
subject matter competence which teachers should have, nine seperate cards 
would be used for recording the information provided by each evaluator type. 
Depending upon conformance to the policies on the minimum number of reports 
required/evaluator type, the system will permit the retrieval of information 
on up to three user type levels (e.g. - past users; past users who are teachers; 
past users who are teachers with an "average" subject matter competence), in 
any combination. This can be achieved by having the standard classification 
entry on each card designate the following Information (# of card columns re- 
quired for maximum needed adaptability also shown): 

of Columns 

1. The product module represented on the card (e.g. - Elementary 

Science vs. Overhead projectots-projectuals) I 

2. The general classes of product information (e.g. - general de- 
scriptive vs. recommended usage) ^ 

3. The major class of product Information within the general class 

B 
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of Columns 



(e*g* teacher infoinnation within recommended usage* 1 

4* The minor class of product information within the major 
class (e*g* - teacher subject matter competence informa- 
tion within teacher information) 2 

5* A value on the rating scale for the minor class of 
product information (e.g. - "average” vp. "secure" 
subject matter competence)* 2 

6* The major information source type (e*g. - past users 

vs* producer) 1 

7* The minor information source type within major (e*g* - 

students within past users) I 

8* The subminor information source type within minor 

(e*g* - subject matter interest within student) 2 

9* The "value" or rating of the subminor information source 

type (e*g* - "low" subject matter interest) 2 

13 columns 

Liberally^ the storage of this information would require 15 columns on the 
cards (minimum « 13)* The remaining 65 columns are available for designating 
specific products. Since there are 12 rows on the cards, there are 12x65 ox 
7S0 row-column intersections remaining to be used in designating specific 
products* Thus, within a product module the system can accomodate up to 
780 products on each card* This will probably meet the product limitation 
specifications to be developed under the guidelines suggested earlier* If 
more than 780 products are included in a product module, a second card can be 
created which duplicates the standard classification information on the first 
card, with product up to #1560 indicated by the 780 row-column inter- 
sections on the second card* The second card would have 
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to be designated as such (e*g* • by putting a (1) in the unassigned column yM5) 
so that the analyst will know that a hole at a particular row-column intersection 
designates a different product than that designated on the first card by a hole 

at the same row-column intersection. 

The card layout below is an example of the design discussed above: 

Card Layout Showing All Elementary Science Products Which Hav e Been Recommended 
Qv Students of Low Subject Matter Interest For Use By Teachers of Av erage Subject 
Natter Competence 



Standard Classification System 



Cfoordinate Index System 



^ Product or Eval- 
uation Info^a- 



Evaluator 
Information 






Products Recommended (By Code #) 






Column # 
1 

2 

3 

4-5 

6-7 



S 



9 

10-11 

12-13 

14-15 
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02 
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Code # 
1 

2 

2 

01 

05 

2 

4 

12 

02 

00 



Code Meaning 

Elementary Science Product Module 

Recommend Usage (Product Evaluation) (1st Level) 

Teacher (2nd Level) 

Subject Matter Competence (3rd Level) 

Average (3rd Level Value) 

Past User (Evaluator) (1st Level) 

Student (2nd Level) 

Subject Matter Attitude (3rd Level) 

Low (3rd Level Value) 

.Columns Ifaa^signed (Put 1 in Column 15 if Need 2nd Card) 

- ^ . 



In other instances, a similar layout will be used to record information 
about produces* For example, one card may designate a specific price range on 
Elementary Science products. If the inquirer stated that he only wished to 
consider products within this price range, the card could also be pulled* 

When overlaid with these cards of the "recommended by - recommended for" type, 
products meeting the inquirers "recommended by - recommended for" specifications 
but outside the desired price range would be blacked out* Regardless of the 
particular information on a card, the same design principles explicated here 
should be applied in carrying out a further analysis of this design alternative* 
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Appendix C: Working Document 



A Systems Design Including an Information Storagg_a n d Sys.t^ 

fo r EPT^‘*s Thr^-Year Development Period *••* Janua^ 196b ^^Decemiber 197^ 



During its development period, EPIE proposes to design and establish 
procedures for collecting, processing, storing, retrieving, and disseminating 
educational product information obtained from producers, analysts, and 
users, and for the synthesis of those three types of informaticn for 

any given product. 



The operational objective for the period is to develop the data 
system in a manner that will enable EPIE to provide services to its 
subscribers first in a broadcast mode, later in"a responsive mode, then 
in an interactive, and, eventually, perhaps, in a customized mode. 



To this end, EPIE must; 

. Test operationally and evaluate the procedyres and techniques 
used in collecting data and revise them as required. 

. Design and operate a storage and retrieval system which will 
facilitate the carrying out of its purposes efficiently and 
at as low a cost as possible. 

. Conduct research on how product information contained in 
the system is being utilized and may be better utilized 
by decision~makers in schools and industry. 

. Experimentally explore and use methods for immediate auto- 
matic servicing of inquiries from subscribers in an inter- 
active mode from remote "satellite" inquiry stations. 

During each year of operation EPIE will invite professional com- 
mittees to audit its information procedures. The audit committees will 
be comprised of outstanding personnel in the fields of education and 
information retrieval, who will provide an external source of evaluation 
and guidance for the conduct of the information effort. An additional 
source of guidance may be explored through four-day symposia conducted 
by EPIE to examine the theoretical and the operational constructs of 

the information. 

The First Twelve Months. The first year of activity (s?e Figure I^^ 
would be directed toward evolving a developmental version of the EPIE 
information procedures. The effort would be undertaken within the 
Four-State Cooperative Project, designed specifically to assess the 
methods whereby EPIE would collect and exchange product information 
among schools in these and eventually other states., Therefore the 
major objectives of the first twelve months include; 

. Pretesting the data collection instruments designed to 
gather product user information. 

, Conducting extensive user interviews and analyses of utilization 
by participating schools in the four states to assess the 
appropriateness and usefulness of the information being 
provided to the user. 



• jk ’®igning and training for the utilization of the initial 
version of the information storage and retrieval system in 
dealing with information gathered from producers, product 
analysts, and product users. 

• Continuing tho broadcast mode of service and trying out the 
responsive mode. 

•• . Building an initional operating version of the storage and 

retrieval system. 

. Identifying inquiries that may make special demands that 
cannot be handled by the system in a responsive mode, and 
developing computer software to handle certain categories 
of these inquiries. 

• Evolving methods for synthesizing information. 

. Providing professional audits of the information procedures. 

The Second Twelve Months. During th? first year of development, con- 
siderable experience will have been gained in following EPIE*s informa- 
tion procedures. While plans for the second year are of necessity 
dependent on the results of pbservations of the first year, it is 
possible to see ahead the broad outlines of what will have to be under- 
taken . 

For instance, in the second twelve months it will be possible to 
include some analyst and user product information in the broadcast mode 
of service, and to initiate for a limited number of schools the respon- 
sive mode of service in certain curriculum and equipment areas. Infor- 
mation on additional curriculum areas and equipment classes — secondary 
school social studies, perhaps, and ei^t-millimeter films — can be 
added to the available fund of data. (See Figure II.) 

To facilitate the increase ip service, extensive research will 
be undertaken during the second year in order to gain greater insist 
into the information needs and use of information on the part of pro- 
duct exchange users. As a corollary to this effort the computer pro- 
gramming designed to handle general requests will be modified to allow 
for integration of special-request software. Simultaneously, attention 
will 1^5 given to the evolution of guidelines for use of the data in a 
humanistic fashion, in both the interpretation and the collection of 
the product data. 

The specific objectives for the second twelve months include: 

. Refinement of data collection and dissemination instruments 
and documents. 

. Collection, storage, and dissemination of information on 
addititonal equippient categories and curriculum areas. 

. Synthesis of the three types of information into product 
profiles. 



. Conduct of research on patterns of information use. 

. Introduction of the interactive mode of service. 

« Investigation of a customized mode of service. 

. Development of software to handle special requests. 

• Development and conduct of a field trial to explore remote 
teletype input-output systems. 

. Evolution of guidelines for humanistic handling of 
information. 

. Conduct of the first Information Symposium and provision 
for professional audits. 

The Third Twelve Months . The principal theme during this time period 
will he one of continuous evolution. (See Figure III.) The focus will 
he upon the expansion of the system, increased user service, refinement 
of the operating system, and intensification of the analysis to determine 
user needs and utilization of EPIE information. The commitment will he 
to create a flexible, adaptable system which possesses sufficient re- 
sources to generate improvement and expansion on a continuous basis. 

An underlying premise of the information effort is that some 
information is of less value to the "EPIE man" in making product de- 
cisions than other information. EPIE is committed to analyzing this 
problem as an integral part of its total effort. The activity relating 
to this area will be increased during the third year. Development of 
methods for bringing the desired information to the user more rapidly 
and in a more usable fashion will be investigated. Efforts will be made 
to increase the number of users of the system through various methods, 
including short-term trial use. This would also serve to test the upper 
operational limits of the operating system. 

The overall objectives of the third twelve months include: 

. Revision of the data collection instruments and procedures, 
based upon the data information research. 

. Inclusion of information on additional curriculum areas and 
equipment classes. 

. Expansion of user information research. 

. Conduct of extensive field interview! with users. 

. Exploration of limited user use of t„e system directed toward 
obtaining more EPIE users. 

. Identification and establishment of new physical methods for 
querying the system and providing responses. 

. Continuation of Symposia and professional audits. 
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Overall Operation of EPIE^s Information Procedures * EPIE*s activities 
during the first year vill be directed toward making its procedures into 
an operating entity. It will be noted from the foregoing that the effort 
for this initial period is focused on the establishment and selection of 
the material to be maintained in the exchange together with detailed pro- 
cedures for operation. During the second and third years the focus will 
be on improving and expanding yhe system. 

Indexing and filing of data obtained form Instruments used in the 
Four-State Project, from reports of professional subject analysts, and 
form producers; retrieval and dissemination of pertinent material in 
response initially to fixed requirements of users and later to flexible 
requirements; and continuous monitoring of the quality of retrievals 
with respect to user requirements as well as suitability of retrieval 
terms, will be undertaken f 

Identification and Selection of Data . The primary sources of data will 
include interview protocols, mail questionnaires, telephone survey forms, 
reports form content analysts, and information from producers of products. 

Operation of the Input System . The input subsystem includes the indexing 
and preparing for data base insertion of that information which has met 
the criteria for incorporation. Development of this subsystem includes 
evolution of thesaurus vocabulary for a gazetteer file and designation of 
the facets of a map file; user profile evolution; procedures to carry out 
indexing, accessions, filing, and dissemination functions; and establishment 
of the output processing and service functions required for dissemination. 

The assignment of retrieval terms will be made by EPIE staff, con- 
sultants, advosory boards, and others experienced with the product classes 
under consideration. 

Products of the System . The EPIE system will be one of a central data 
reference and control, serving to locate and retrieve the pertinent and 
appropriate data in usable form. Some of the prime outputs and products 
of the system include; 

. A user product description with various levels of data 
related to the user request, which might include prior school 
use, students and teachers involved, grade level suitability, 
classroom'use conditions, content strategies, employed, etc. 

. Data synthesized in such a fashion that it may be used in 
periodicals, monographs, etc. 

. Tables, matrices, etc., showing relationships between users 
and data requested. 

. User profile feedback forms, used by the receiver of the 
documents to evaluate the quality of retrievals against his 
request. 

. Regular user profile analysis data, providing a record of 
those terms in the vocabulary and user profiles which have 
been successful in providing retrievals, or have formed the 
basis for dissemination against profiles and requests, and 
o ■ • - 
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making possible the evaluation of the vorth of key words used 
in the system. 

• Management control reports for evaluation of system statistics. 

By keeping track of data entered^ numbers and identification of 
those distributed to different types of users, and othQr statistics 
of operation of the system, a continuing measure for evaluation of 
the operation is available . User data cost anW.ysis. could be 
readily extrapolated from this system. 

User Profile Development and Dissemination Techniques . Library e>^erience 
has shown that regular dissemination of documents of interest to specialists 
in a field can best be accomplished on the basis of user interest profiles. 
The steps are, briefly, as follows; 

. Insure that the user has been given a clear picture of the 
overall design concept in which his profile will be operating 

, Describe to the user the contents of a profile and how it 
will be used to provide him with relevant materials . 

. Assist the user in constructing a list of terms by providing 
him with a thesaurus as a guide, but do not restrict him to those 
terms should he feel others to be more relevant. 

. Provide a test period during which data are provided to the 
user on his profile. 

. When the system is operational, prepare and deliver to each 
regulnr user a periodic profile review and analysis. 

Oneration of the Storage a nd Retrieval System . The initial design and 
development of the storage and retrieval system will be undertaken with 
the assistance of an organization which has had considera le experience 
in designing and developing computer-based information activities. 
Investigations will be made of the possibility of obtaining existing 
programs which can be utilized in EPIE*s system with few modifications. 

Thus far in the design of its information system EPIE has contracted 
with Community Systems Foundation, Ann Arbor, Michigan, for the creation 
of a working document describing an*'BPIE. information system. Since the 
completion of that document EPIE has also received consultation on its 
systems design from System Development Corporation, Santa Monica, California. 

Gradually as the system evolves, the contracting systems organization 
will train EPIE personnel in the procedures and techniques required to oper- 
ate the storage and retrieval system. The organization would continue to 
consult during the three-year efiort and would be responsible for the design 
and development of the special requests software. By the end of the third 
year the specialized organization'* s role would be almost entirely con- 
sultative • 
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Appendix D 



Draft for Discussion; 

Curriculum Analysis 

for EPIE Work Conference, Dec. 11-14, 19^6 

Ira J. Gordon 

One of the main problems curriculum people face is to create some 
internal consistency in their programs. To do so the theory of instruction 
inherent in the marketable supplies and programs being suggested must be 
analyzed. These offereings must be able to be examined so that coiopletely 

disparate systems will not be used unknowingly. 

Any brief review does not do justice to the programs discussed nor are 
they assumed to be typical curricula for all children. The purpose is to 
deomonstrate the use of the criteria to assess materials t 

A way of approaching materials is to ask; what assumptions, postulates, 
or hypotheses are evidenced in the material concerning; (l) pupil character- 
istics as (a) the nature of the child, and (b) the nature of how children 
learn or how learning occurs; (2) instructional situations as (a) the 
sequencing of es^erience, (b) the selection of didactic materials, and (c) 
the nature of interpersonal relationships; (3) goal characteristics, what are 
both the immediate and long range goals and purposes. 

In order to test whether these procedures are usable three different 
areas have been selected and particular projects identified within the areas. 
In the field of "the new math", we \>rill look at Robert Davis’s M^isqn 
Project, an integrated systematic approach. Second, from the sciences we 
will look at the Biological Science Curriculum Study . Third, fromtthe area 
of social studies we will examine one of the projects in the Developmental 
Economic Education Program sponsored by the Joint Coincil on Economic 
Education. These samples may not be typical and shoild be considered 
merely as illustrative. 
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I, Pupil characteristics 
Exan^les : 

A. Robert Davis’s Madison Project in Mathematics is one 

example of a new curriculum. 

"...every effort should be made to get the students 
thinking and talking, not listening and accepting*" 

(Davis, 1964a, l8). 

"(Children) learn because, on the one hand, they are 
happy, and on the other hand, intellectual challenges 
are abundantly present in many attractive forms." 

(1964a, 7). 

B. BSCS 

(Ed. Note: While the nature of the child and of how ' 
he learns is discussed briefly, chief emphasis in Gordon’s 
discussion is given to the process by which the learning 
occurs. It is noted below.) 

The process is one of enquiry and discovery. There are demonstrations 
and observations, experiences in participation. There are 12 character- 
istics which might be thought of as indicative of a discovery approach. 
These are: 

1. Concepts are presented by the inductive method. 

2. Concept presentation is initiated by a problem situation. 

3. The materials encourage active participation of the leeurner 
in textual as well as laboratory investigations. 

4. The materials are structured to raise questions in the 
mind of the learner. 

5. The structure demands that the learner organize his 
knowledge. 

6. The structure encourages the learner to ask questions 
utilizing previous information. 

7* The materials present the student with new information. 

8. The learning environment is manipulated, not the learner. 

9. The structure allows for differences in the student’s 
level of cognitive development. 

10. The generalization to be learned is withheld from the 
learner so that he can discover it for himself. 

11. The "discovery" made by the learner is not necessarily 
new to civilization; it is new in relation to the previous 
cognitive development of the learner. 

12. The investigative procedures used are open-ended; the 
teacher does not know all possible results and explanations 
which may grow out of a presentation. 
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C. DEEP 

The statement of ’ goals for the elementary level gives an indication 
concerning the nature of the child. The authors state: "The main purpose of 

economic education in a free society is to develop the problem solving ability 
of our children as it relates to personal and social problems, basically 
economic in nature. To accomplish this purpose we must: 

1. develop the child’s analytic ability 

2. help the child relate his everyday experiences 
to the big world around him 

Even a first grade child has economic decisions to make." (1964, 1-2) 

Several inferences (about the nature of learning) are possible: 

(1) learning occurs best under discovery and problem solving conditions; 

(2) particularly for younger children, first-hand direct e3q>erience and 
personalized meanings are impoilant; ( 3 ) learning problem solving per se is 
a basic part of learning economics. 

It is clear that Davis sees the child as a cognitive information processer. 
He is concerned with the development of concepts and he feels that intro- 
duction of abstract experience should be earlier in the grades. There is the 
assumption that readiness builds on earlier experience and that this ejq>erience 
in mathematics, at least, must be encomtered earlier. 

The notion of positive reinforcement permeates the thinking behind this 
project, although Davis does not say so, with the teacher actively encouraging 
the child’s responses. 

It is, however, important to know that the teacher’s response is to be 
gi<,ded not by the correctness of the answer but by the spirit of encouraging 
the child to continue to try. 

The child in the Madison Project classroom is expected to utilize trial 
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and error learning as a part of the discovery concept. 

In place of demonstration and observation, Davis stresses ej^erienpe 
and participation. Indeed, if there is a key concept in his approach it is 
this ambiguous notion of experience. E:q)eriences are often seen as the 
end as well as the means. 

In BSCS, the child is seen as an information processor and an enquirer. 

But, what a chil4 learns is not necessarily a function of what he waists to 
learn or what is useful to him. The concept of needs used by BSCS are all 
in the cognitive domain. 

Neither cognitive dissonance nor positive reinforcement are clearly 
indicated. Indeed the position is unclear, except for the vague conception 
of "discovery” and activity. . . 

An analysis of the text and laboratory indicate that they make extensive 
use of presenting concepts by the inductive method, in initiating concept 
formation by a problem situation, by presenting concepts through active 
learner participation, raising questions, and presenting new information. 

The analyst states the laboratory has a total effect of moderate accoroplish- 
ment of the methods of discovery learning. 

Words are inportant in biology both in its classical taxonomic forms to 
which we were subjected by as students and in its modern conceptual design as 
presented by BSCS. Labelling, categorizing, naming are all very much part of 
the game. Groip discussions help, however, and seem to be indicated in the 
various materials developed by BSCS. 

There is focusing on relevant mr'.eyials. However, the focusing in 

BSCS seems to lack sharp definition. The principles upon which students 

are to focus are also at rather high levels of abstraction and it may be that 

exercises in which youngsters engage are several steps removed from the principl 
o 
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The main source of data in the Developmenta l Economic Education Pro6r?m 

« 

is The Joint CouncU on Economic Education Program, puide to Develop- 

mant.Ai p<».onomic Education Program, suggest i ons for grade placement _a nd ^evelg E:: 

insiit of economic ideas and con.CGP’fcs * 

The child is seen as a problem solver, and, more than that, someone who 

is concerned with utilizing academic concepts to handle his daily needs. A 
cursory view of the various ideas ?nd concepts to be developed at elementary, 
junior, and senior high level indicates a belief in development of the child 
from someone who can deal with materials only at a concrete, highly personal- 
ized, immediate perception level to a youngster who can relate abstract con- 
cepts to other abstract concepts, who can analyze "big ideas," who can engage 
in the utilization of abstract sjimbols and who is able to handle a variety of 
equations utilizing abstract labels. EssentiaUy, the cognitive domain is 
stressed with effect and personal involvement diminishing as one moves 

the developmental scale in this guide. 

For younger children role-playing, dramatization, puppets, resource 
people, parents are all utilized to provide e:qierienoes in illustrating the 
economic concepts. Problem solving is seen as a shill to be exercised at 
every grade level and the steps ai-e the symptoms of the problem, certain 
aspects of the problem, causes of the problem, and solution to the problem. 

II. Instructional situations 
Examples : 

In sequencing in the Madison Project, 

"children are asked a carefully devised sequence of questions, 
which gradually lead th^ to formulate tectaiques of solution 
and to discover generalizations." (l9o^a<} S). 
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’’Cues and answers are 
discover the internal 



withheld to allow the child to 
organization of the subject." (l9o4a, 9 ) 



The didactic materials serve as self ipeaching devices. 

For interpersonal relations, "...It is for the child to 
decide what attitude he will take toward h^S but . 

their attitude toward him is to respect him a$ ^ equal, and 
a friend." (1964a, 6,7). 



In BSCS, whatever sequencing there is seems to be under the control 
of the teacher, but Just how this control is exercised is not clear. 

The student uses the didactic materials in controlled situations to 
develop or to discover for himself the existence of these principles, rather 
than having them stated in, to use Ausubel’s term, "advanced organisers" 
for him. Through the manipulation of the didactic materials, he is expected 

to develop his concepts. 

It is in the area of interpersonal relations that BSCS gives no informa- 
tion beyond the idea that the teacher is to encourage and challenge. 

DEEP, with the spiral curriculum developed around basic concepts, has 
concepts increasing in scope and complexity, depending upon the understanding 
and economic sophistication of the pupils. Teachers should plap to reinforce 
at higher grade levels concepts introduced at the lower elementary level. 

Didactic materials are sued in mys similar to the following example. 

"To illustrate how instability occurs in our economy the child can drama- 
tize the production of various articles for a given year. After the dramatiza- 
tion, children will place op the table toy models of what they have produced. 
One person, who is the chairman of the group, ^dll purchase all of these 
goods and services with toy money. The toys piled on the table represent 
the GNP. Then the teacher will explain that whether industry continues to 
produce or not depends upon the intentions of the consumer, business and 
Q“overnment, and foreign countries.' 
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In the area of interpersonal relationships, there are no direct suggest- 
ions in the guide for teacher-pupil relat^onst^ip or for peer relationships. 

Davis defines a teacher as a moderator or a discussion leader as one who 
asks a series of leading questions and as one who observes and listens to 
children. The teacher plays a central role as the organizer of e^qperience, 
although this seems to be more an art than a. prescript! on. 

Although the child may have the impression that the experience is 
haphazard, and although there is no real effort to strengthen the sequence 
and eliminate all "noise", nevertheless the sequence is determined by the 
structure of the discipline. This is what the child is to discover. 

The teacher is not to give answers or let the child know exactly what 

is expected or to be discovered. 

There are no concrete materials utilized in the sense of the manipulation 
of objects. However, materials are concrete in the sense that the child 
can assess his answer against whether it works or not, rather than raljring 

v®on authority judgment from the teacher. 

There are positive injunctions in the Madison Project for how the 
teacher ds to behave. The teacher is an encourager and respecter of the 

child, providing a warm accepting climate. 

In BSCS, there is obviously a conflict among the biolpgists as to 
which basic principles to center upon and what ordering of materials is 
desirable. The existence of (...differing) versions in which sometimes 
the same materials are used but appear in different orders, would certainly 
indicate tremendous research possibilities for the investigation of appro- 
priate sequencing of materials in biology. It mi^t also be indicative, 
subject to research, that sequencing really doesn't matter. 
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The laboratory materia? p particularly seemed to have been selected 
with a notion of what is necessary and sufficient for concept attainment. 

The student deals with real objects and he tests principles. He finds out 
and. develops his notions from vdiat happens to these materials as he works 
with them. They do not seem to be selected to represent fixed answers to 
fixed questions. 

There seems to be no understanding of the role of affect in learning 
or any concepts of classroom climate. There might be great discrepancies 
between what is written in the textual materials and the mood and spirit in 
which life is lived in the classroom. This is not m^ant as a criticism, since 
this was not a key focus for BSCS. It indicates that it has inconT>lete 

asBunptions about classroom behavior and learning. 

In DEEP, concepts are dealt with through problem solving analysis at each 
grade level with a movement from concrete to abstract- symbolic, from personal- 
ized effective loading to depersonalized rational analysis. 

It seems as though the didactic materials from junior and senior high 
school are all words whereas in the elementary schools they may be such 
things, as, in teaching competition, setting up a lemonade stand and selling 
lemonade, with the second child cutting the price or pretending to. In the 
elementary grade, it is suggested that all play certain jobs to illustrate 
the importance of division of labor. There is also the utilization of films, 
filmstrips, photographs, painting materials, toys, and other objects that 
may illustrate particular concepts. 

The materials used directly relate to the concepts to be learned, offer 
c I'^ren opportunities for direct perceptual experience which can lead into 
discussion and elaboration. 

(In the area of interpersonal relations, Gordo, indicates that some 
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suggestions for elementary children seemed somewhat naive about how 
children relate to each other when they are other than middle class child- 
ren or even just children. Two examples were cited.) 

III. Goals 
Examples: 

In the Madison Project, a key purpose is to immerse youngsters in 
mathematical experience so that they will become familiar with the concepts 
indirectly, and so that they will then be able to utilize them without 

necessarily being conscious of their knowledge. 

In BSCS, perhaps a very brief line from the Teachers Commentary will 

indicate the goal: 

"A sound sense of the nature and values of science, may lie 
in the new responsibility of science teaching," (19^0, XIIl). 

To e 2 q)licate this, scientific knowledge is analyzed for its utility. 

Two project examples are cited because of their diverse approaches 

to goals. 

The Pittsburgh program, for twelfth graders (19^5 ) sees children and 
learning in terms of problem solving on topics fraught with personal mearing. 
Although it deals with concept development, it recognizes the concepts have 
affective loading. 

On the other hand, the Sapta Rosa material stresses a highly analytic 
cognitive attack. In one lesson for twelfth graders, it is stated, "This 
lesson is largely inductive. . Ihe first discussion, after filling in the 
chart, should allow the claps to induce some of the generalizations suggested. 

Nowhere is ib clear that children are ever asked, in the Madison Pro- 
ject, why they should wish to learn whatever is being presented. It is 
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assumed that curiosity is a sufficient beginning motive and that competence 
will take care of itself, although certainly competence is a goal. There 
are no concepts to transferability, and what is to be discovered is to some 
degree left to chance. There is further, to use Davis' terms, no "speci- 
fied level of achievement." In this respect, what is learned by each 
youngster is highly individual and Davis is not concerned with verification. 

Major concepts are that the notion of fixed maturation must be called 
into question, and that instruction or experience qf particular kinds may 
very well lead to modification of cognitive structure. 

It is clear from BSCS material that the major goal is a content goal. 

It is not to change the learner, or to hove him feel that biology is fun, 
or to develop his general intellectual capacity. The main goal is to enable 
him to learn the structure of the discipline of biology, in effect as a 
biologist learns it. 

One would suppose a major goal is learning to learn, that is learning 
how to enquire so that one may continue to enquire throughout his life. 

There is some notion of application to social probleras, but certainly the 
didactic materials and textual materials do not indicate ariy way in which 
biology is transferable to such problems as conservation, water supply , 
and air pollution, for example. 

When we come to goals the DEEP guide is very clear. There are the 
concepts mentioned under sequencing to be learned, there is the emphasis on 
teaching children to solve problems, to thinlc in economic terms, to be able 
to perceive relationships between history, geography, economics and other 
social studies, and to relate to the world. The DEEP materials much more 
directly relate to their goals of citizenship than do the BSCS materials. 
Although obviously there are economi'; concepts to be learned, there are 
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also consistent attempts to apply these to the life of the student. 

It is to the credit of the Joint Council that two such diverse pro- 
graiiis as those in Pittsburgh and in Santa Rosa can epierge from its guidelines 
but it certainly reflects differences in assuraptj.ons about the nature of 
children, the nature of learning, the goals, the seleption of didactic 
materials, and what might be called a theory of curriculum development or 
a theory of instruction in the field of concept attainment in the area of 
economic concepts. 



Conclusion 

I have atteaptfid, not to analyze the projects, but to discuss whether 
the criteria can and should be used this way, and whether this demonstration 
is valid. 
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SECOND LAKE MOHONK WORK CONFERENCE 

April 8'«'10, 1967 



Present: Abraraovitz, Baranski, Denny, Easley, Filep, Goodman, Gordon, Harris, 

Komoskl, Lockhard, Morrissett, Odell, Preston, Stake and Stevens. 



PKK introduced new participants to the group, and stated the purposes of the 
meeting : 

1. To design and define the pilot study in a four*~state area, and 
to draft a Title V proposal for the four states. 

2. To define as fully as possible the techniques to be used in 
analysis of elementary science material. 



The group to work on the first objective to consist of Komoski, chairman, and 
include Denny, Preston, Stake, Filep and Harris. The recond group to be chaired 
by Goodman, and include Morrissett, Stevens, Baranski, lockhard, Gordon, Odell 
Easley and Abramovitz. 



CURRICULUM ANALYSIS WORK GROUP 

FG opened the meeting by seeking to establish what the exact goal of the group 
should be. It was agreed that it should be a document spelling out as far as 
possible the variables to be used in logical analysis of educational materials, 
and guidelines on applying these. They felt that though this document should 
apply to elementary science materials, it should be as general and universal 
in application as practical, and relate to both books and non-printed materials. 
Some testing of schemes would be done on the Harcourt, Brace & World series. 



Consultants discussed their various outlines for analysis and differences 
showed up in definition of terms, stress on areas of concentration, and units 
used for investigation. However, there was complete agreement on 
of variables to be used. Emphasis was laid by all on the need for flexpilii-y 
and open-endedness in any system developed, and the importance of revising and 
amending lists as data come in from users. 



In answer to questions abojt actual use of a storage and retrieval system, FG 
explained that any system used by EPIE would be able to accomodate inputs ^ 
various forms, from checklists to paragraphs. It was pointed out that checklists 
make information comparable in a way reviews do not; however, some infomation 
is difficult or impossible to fit into checklists. The information gathered 
by logical analysis will be only one type of information in the system, and wi 
be supplemented and checked for reliability by information from producers an 

user reports. 



The question of unit of analysis was discussed, as each consultant had a different 
method of approaching material to be investigated. The purchasable unit has been 
chosen by EPIE as the basis for its answers to queries, and tnis varies from quite 
small (one booklet) to large (a whole series) . Material will have to be reported 
on in small units, and also in large assemblages. 
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Consultants discussed what, and how much, information would go into reports. 
Reports would be written, using all information in the system in whatever form, 
then in answer to queries’, summaries to the user would ])e supplied, oriented 
toward his specific needs. 



Purchasing and decision-making procedures were taken up. The group felt that 
not enough is really known about this now, and that the pilot study should 
clarify to some extent who makes decision, on what information, and what the 
schools really want from EPIE. The ERIE questionnaire should provide useful 
data on practices in New York and Pennsylvania. EPIE’s consultants are making 
assumptions from the university level, may not be applicable. It will be 
necessary to choose variables for consideration according to their best judgment 
and. see how these fit users’ expressed concerns. It is believed that information 
will only be desired on new materials, copyrighted within the past two years. 
Important to think of users' needs, not general academic view of textbook. 



As soon as any information is in the system EPIE can start reporting it, and add 
more as it comes in from other sources. Probably producer descriptions and claims 
will be the earliest inputs, with logical analysis and user reports coming in later. 
User information will not be reported out until there are enough replies to give 
reliability. EPIE will review and analyze supplementary materials as well as 
textbooks. At the center at the University of Maryland, their main concern is 
with hardware and software, not books. Teachers need information on new materials 
even more than books. They are also interested in the amount of training necessary 

to use them. 



Questions were raised about the dependability of inferences made by analysts. 
These may vary greatly and should be documented where possible from the material; 
the analyst's identity and background should also be noted. 



The group discussed how to start working on forms and training of analysts; a 
timetable is being worked out. Gordon, Easley and Lockhard are sources for avail 
able manpower during the summer months, some free. 



Questions were asked about the usefulness of the ERIC thesaurus and Bloom s taxonomy 
as a base for lists. FG explained that words were only put into the tnesaurus as 
materials using them came in, and there were f'jiw science materials so far. He t 
that it would be a good resource; Bloom's taxonomy would also be good, though not 

useful for retrieval purposes. 



Comoatabilltv of reports was discussed again. The group agreed that analysts must 
be free to work in Afferent styles, that different points of view would give more 
information to system. Information would be pulled 

cards and coding were discussed, and the relative merits o.. direct 1= 

coding explored. Direct coding better for computer systems, but inverse coding is 
a valLble tool for suggesting possibilities. Can use direct cards for inverse 

sorting . 

Mechanical means for getting into material were discussed, such as KWIC index, 
classifying all questions, or activities, etc, to give analyst a way to start. 
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Group tried out systems on BH&W material, with favorable results as to feasibility 
of use and general agreement on evaluations* Both text and supplementary kit 
were examined. 

Consultants agreed that EPIE system of analysis must be composed of several 
systems. At the end of meeting they submitted papers outlining their systems and 
suggested guidelines for conducting investigations, which could be used in training 
other analysts. FG prepared an overall summary report of all systems. 
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Appendix F: Working Document 



J. A. Easley 
April 1967 



Plan for EPIE Elementary Science Series Analysis 



Introduction; 

It is proposed, during the summer of 1967 (June 15 - August 15), to 
analyze at least three or four elementary science series with auxiliary teaching 
materials at the University of Illinois. During the first week, we will orient 
the team of analysts to the sets of descriptors required for the analysis and to 
put the final touches on the lists of descriptors and the forms to be used for 
recording purposes. The analysis will work from the rationale outlined Easley 
at the Second Mohonk Conference. It is expected to be similar to the system 
he used to analyze contemporary standardized biology tests (A “Bio Assay 
of Biology Tests). The sets of descriptors used in this project will be developed 
especially for elementary science series. However, the summaries of the 
analyses, which will be prepared for each purchasable unit, will be directly 
relatable to the general-purpose outlines prepared by Gordon and by Morrisett 

and Stevens . 

Analysts: 

The team that will be used to develop and carry out the analysis procedures 
this srnnmex will include three members of the Science Department of Univer- 
sity High School, one of whom (Kendzior) was involved in the Biology test 
project. It will also include a staff member (Guthrie) of the Training Research 
Laboratory (TRL) who is experienced in the analysis of conceptual tasks . 

Easley will direct the operation and do much of the writing of summary reports. 

Planning Discussions ; 

Before and during the two months, the analytic techniques will be discussed 
from theoretical as well as practical points of view. The following persons 

are expected to contribute to the discussions: Hastings, Atkin, Stake, Glass, 
Millman, Dem y, Shoressman, Gould, . Payette, McGuire and Anderson. Two 
of these persons will need to be employed as consultants in the early part of 
the two -month period. 
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Elementary Science Materials : , 

The four series to be analyzed will be selected so as to ensure early and 
optimum access to user information as well as to test the analysis system by 
a broadly representative sample of materials. It is expected that the materials 
themselves will be available through the University’s Curriculum Library. If 
a set is desired for analysis which is not available here, other means of 
obtaining it will have to be found. 
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Code for Location of Assignable Unit 

Publisher Abbreviation, Year of Publication, Grade Level, Chapter, Page(s) 
e.g. Silver Burdett 1965 6 19 234-5 

beconies: SB-65-6-19-p^4-5^^^^. When the assignable unit is found in the Teachefs 

Guide only (TGO) its location on the page will be found 



on the left of the space in 0*s. 



Code for Format of Assignable Unit 

Information about student pages is found to the right hand side of the space 
provided on the analysis sheet. 



Includes 



• f 



Text T 

Question (s) Q(No.) 
Activity A 



Photograph, color 


P.Ctio.) 


black and white 


Drawing or Diagram 


• 

o 

u 


D^(Ho.) 


Chart 


Cg(No.) 


C^(No.) 


Graph 


G^(So.) 





Activity Located Only in Teacher’s Guide 



P|,(No.) 



TGO 



I 



e.g. An assignable unit with some textual material, three questions, one 
black and white photograph and two color drawings. 



becomes: T, Q3, P^, D^2 

An assignable unit with some textual material, an investigation, one 
question and a chart in color 

.becomes: T, A, Q, C 

An assignable unit which is an activity suggested by the Teacher’s Guide 
but not mentioned in the student text 

become-s: TGO (This will be located on the left hand side of the space 

' provided on the’ analysis sheet) 
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Adverbial Descriptors 



A numerical scale from 1 to 5 is used where the numbers represent terms 
in the following lists* 



!• Impressively 
Outstandingly 
Strikingly 
Skillfully 
Cleverly 



2* Clearly 
(Completely 
Effectively 
Sufficiently 
Interestingly 
Appropriately 



3* Adequately 
Suitably 
Satisfactorily 



4, Routinely 
Commonly 
Unclearly 
Incompletely 
Ineffectively 
Insufficiently 
Inappropriately 



5# Erroneously 
Poorly 
Weakly 

Stereotypedly 

Boringly 



Host often these words are used to mean Excellent^ Good^ Average^ Poor and Very Poor^ 
(in order from 1 to 5) as they are actually used in practice. 



Methods of Presentation 



Asks Question (s) (AQ) 

Compares (Com) 

Defines (Def) 

Describes (Des) 

Explains (E^P) 

Gives Directions (GD) 

fftteuextomxK 

Lists 

Mentions (Men) 

Presents Historical Background (PHB) 

Presents Information for Analysis (PIA) 
Presents Model (PM) 

Presents Theory (PT) 

Presents Vocabulary Exercise (PVE) 

Suggests (Sug) 

Summarizes (Sum) 

Tells Story (TS) 

Uses Analogy (UA) 

Exhibits or Demonstrates (EXH) 

Leads Discussion (bD) 

Leads Field Trip (LFT) 
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Student Task Expected by the Assignable Unit 



Apply Knowledge to New Situation (AKNS) 



Calculate (Calc) 

Comprehend (Comp) 

Consider (Cons) 

Construct Chart (CC) 

Construct Graph (CG) 

Construct List (CL) 

Construct Model (CM) 

Construct Picture (CP) 

Decide (Dec) 

Describe (Des) 

Exp lain (E^cp) 

Hypothesize (Hyp) 

Interview Authority (lA) 

Invent (I*'^) 

Investigate ("Experiment”) (Ivst) 



keep a Record 

Look at Suggested Readings 

Make Collection 

Memorize 

Observe 

Organize 

Plan 

Predict 

Present Demonstration 

Present Report 

Read Reference Materials 

Recall 

Recognize 

Review 

Take a Field Trip 
Wonder 



(KR) 

(LSR) 

(MC) 

(Mem) 

(Obs) 

(Org) 



(Pre) 

(PD) 

(PR) 

(RRM) 



(Reco' 

(Rev) 

(TFT) 

(Won) 



* 

Image of Science Given by Assignable Unit 



^Science is" 



Attacking Problems 

Classifying Things 

Controlling the Environment 

Cooperative Human Endeavor 

Creative Human Endeavor 

Curiosity 

Discovery 

Exploring 

Facts 

Qeesing Things 
Ideas 

Learning about Myself 
Making careful Observations 



(AtP) 

(ClTh) 

(CTE) 

(CoHE) 

(CrHE) 

(Cur) 

(Dis) 

(Expl) 

(GrTh) 



XLaM) 

(MCO) 



Making Measurements 
Making Things 
Messing About 
Nature Study 
Openmindedness 
Persistent Human Endeavor 
Specialized Language 
Taking Things Apart 
Technology 
Verifying Things 
Working Carefully 
Mysterious Phenomena 



(MM) 

(MTh) 

(MA) 

(NSt) 

(Ope) 

(PHE) 

(SpL) 

(TTA) 

(Tech) 

(VTh) 

(WC) 

(MP) 









Mode of Assignable Unit 

joBaiixtecficiaacsc^^ 

Behavior- activities of individuals, 
action of any material. 

Classification- assignment to groups within a system of categories distinguished 

by structure, origin, etc. 

composition- manner in vhich parts are combined. the constituents, 

organization or grouping of the different parts. 

Development- growth or expansion. 

going through a process of evolution.. 

Distribution - places where things of any particular category occur. 

location. 

I 

Existence- state of being. 

Interaction - action on each other. 

Origin - that from which anything arises. 

' the source. , • 

the first stage of existence, the beginning. 

Process- systematic series of actions directed to some end. 

a continuous action, operation, or serxes of changes taking place xn 

some definite manner, 
action of going forward. 

I Structure- arrangement of parts, elements, or constituents, 
an organization. 

) "Transformation- change in appearance, nature, character, form. 

change to another substance. 

) Function- to serve, operate, carry out normal work or activity, 
the use« 

) Relationship- a particular connection 
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UMJT SEVEN: PLANTS AND WiORE PLANTS 
Section 5: Grass Plants from Seeds 
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CONCEPT 

Organisms (living things) reproduce their own kind. 



LESSON 7, page 70 



SUBCONCEPT: Seeds produce the kind of plants from 
which they came. 

Aim of the Lesson 

To provide opportunity for children to observe how grass- 
plants grow; to help them identify cereal plants as grasses. 



introducing the Lesson 



required: a complete grass plant, with roots visible in a 
glass of water (if possible, secure a plant in flower or seed 
stage); hand lens magnifying glasses. 

Let children take turns using magnifying, glasses to ex- 
amine the parts of the grass plant. Tell them where the plant 
was growing when you dug it up. Lead the class into a brief 
discussion of grass, which grows in so many different places. 
Encourage them to wonder about: 

How do grass plants get started? _ 



4. Have the children study the picture at the bottom of the 
page to And out what has happened to the seeds. Help them to 
read the question: What kinds of plants are growi/ig? 

if no one notices that all of the plants look somewhat 
alike, guide them to this observation by direct questions. 

As the observations are made, explain that oats, com, 
and grass belong to the grass family. Encourage children to 
compare the leaves (or “blades") with those on the bean 
plants growing in the classroom (or in Lesson 1, page 62). 

Children will probably mention other kinds of grass plants 
whose seeds are commonly used for food: rice, wheat, rye, 
barley. Invite different pupils to tell of foods they like that 
are made from some of these "grass seeds. 

Develop the understanding that plants and seeds of the 
grass family are important sources of food. 

Extending the Concept 

Through Investigation. Get seeds of cereal plants (oats, 
com, wheat, rye) from a feed store. Let children sow them 
on damp sponges, as shown on page 70. 

I'hrough Key Concept Words. Add grass to the Science 

Vocabulary Chart. 

Through Activity. Put a handful of wheat seeds (or 
cracked wheat from a health food store) between two layers 
of clean cloth on a smooth stone or concrete base. Demon- 
strate how to use a hammer to pound the seeds. Shake the 
n»vaJ through a fine sieve (or flour sifter), and let children see 
that a kind of flour can be made from seeds. 



Developing the Concept 

(by emphasis on the subconcept) 

1. Direct attention to the first sponge in the top picture 
on page 70. Help children to read oats on the card standing 
in the sponge. Call on someone to tell what has been done 
with the oat seeds. Ask what will happen to the oat seeds 
when they get wet. Encourage children to reason that the 
wet seeds will sprout and then grow into new oat plants. 
Discuss, foods (oatmeal, dry oat cereals, etc.) that are made 
from the seeds of oat plants. 

2. Next, ask what kind of seeds are being "planted" on 
the wet sponge in the middle of the pan. Ask what will hap- 
pen to the corn seeds that are "planted in the middle 
sponge. Children will reason, from experience, that the corn 
seeds will grow into new corn plants. 

Invite the class to tell about various corn foods they like 
to eat. Ask what part of the corn plant we use for food. 

3. Then direct attention to the third sponge at the top of 
page 70. Ask for a volunteer to read the label by the sponge 
and tell what kind of seeds are to be "planted" on that 
sponge. Help children to understand that these seeds are 
the kind we use to grow lawns. What kind of plants will 
grow from the grass seeds? 

Through discussion of the top pictures, guide children to 
the following assumptions: 

When the seeds gel water, they will sprout and grow into 

new plants. 

Each kind of seed will grow into that kind of plant. 
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What kinds of plants are growing? 



70 



o . 

“ERIC 



CONCEPTS IN SCIENCE; 1 



UNIT SEVEN: PLANTS AND MORE PLANTS 
Section 7: New Plants from Dispersed Seeds 
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CONCEPT 

Organisms (living things) reproduce their own kind. 

f :*.r .r.*.*" n.c:r t.:x 4^. * y 



LESSON 9, page 72 

SUBCONCEPT: Plants are dispersed to new environments 
by means of seeds. 

Aim of the Lesson 

To provide children Nvith enough experiences to realize 
that new plants are started in new environments from seeds 
that are carried by animals and other agencies such as wind 
and water. 
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Introducing the Lesson 






Take children on a short field trip to discover plants grow- 
ing in. many kinds of places. Call attention to seeds and 
gras, s in vacant lots, between rocks and cracks in pave- 
ment. and in other areas where obviously no one took the 
trouble to plant them. Lead children to wonder: 
j Why do 80 many kinds of plants grow in so many 
I places? 






Developing the Concept 

(by omphasis on the subconcept) 

1. Direct attention to the top picture on page 72. Call on 
someone to teli the story of what the sqrirrel is doing. Pass 
acorns around the class if specimens are available. To bring 
out the essential features of the picture, present questions 
about the sourer of the acorns, their relation to the tree, 
the time of year, and why the squirrel is burying the acorns. 
Develop these ideas: 

Acorns are seeds of the oak tree. 

Squirrels use acorns for food. 

In the fall squirrels store acorns for winter use. 

• Some of the acorns buried in the ground remain there be- 
cause of frozen ground, snow, or even forgetfulness on the 
part of the squirrel. 

Now challenge the class by presenting this question: If 
some of the acorns stay buried all winter, what hap- 
pens to them when the i :ow melts in spring, the 
ground thaws, and the water soaks into the soil? 

To verify answers, refer children to the seedlings in the 
top picture. Identify the seedlings as young oak trees, and 
check for comprehension by presenting questions such as: 
Why are young oak trees growing here? 

What are the new oak trees growing from? 

Can you explain how the acorns got “planted ? 
Encourage answers that reveal the relation between seeds 
(acorns) buried by a squirrel and new oak trees. 

2. Then direct attention to the picture .at the bottom of 
the page. Call on individuals to tell what Is happening in the 
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left.purt of the picture. Try to bring out the following ideas: 

The burs sticking to the boy’s clothes arc seeds. 

The boy pulls off the seeds, which fall on the ground. 
Some of the seeds grow into new plants that have burs. 

The plant at the left of the picture is a full-grown plant 
with seeds (burs). 

3. Almost every child will recognize the dandelion, and 
that the wind is blowing seeds from the head. Ask. Do you 
think that ner-' dandelion plants will grow near here. 
Do you think anyone planted dandelion seeds where 

these plants are growing? 

Through review develop the following understandings: 

New plants grow in new places from seeds, that are car- 
ried there by animals, by people, and by the wind. 

Each plant is like the plant on which its seed grew. , 

Extending the Concept Elsies’ . 

jtirough Investigation. Collect several intact dandelion 
seed hrads. Pull out the "parachutes" and let children ex- 
amine them through a magnifying glass to discover why the 
seeds float so easily on the air. Count aloud with the children 
as you remove the seeds. What would happen if every 
seed isrew into a new plant? Whv do some not grow? 



eCQ grew UlfcU liCTV ri- 

Through Key Concept Words. Add tree to the Science 
Vocabulary Chart. 
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What is happening? 
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)lants. 
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animals 
other animals. 
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animals eat both plants and animals. 
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Discussion Questions 

!• Do you know what kind of an 
animal (page 26) this' is? What is it 
eating? A prairie dog; it is eating a 
dandelion, (note: Have children 
guess the appro.ximate .size of this 
animal by comparing it with the 
dandelion.) 

2. What does the Alaskan brown 
bear have in his mouth? Fish (sal- 
mon) 

3. Do you think the Alaskan brown 
bear eats many plants? No; because 
of cold climate and rocky terrain, 
few plants grow in the region where 
t^ese bears live. Hence, their diets 
are confined primarily to fish. 

4. What is the name of this animal 
that looks as if he is wearing a mask? 
What kinds of food does he eat? A 
raccoon; he eats nuts, seeds, small 
mammals, and birds. 

Special Activities 

1. Obtain a grasshopper and place 
it in a maraschino-cherry jar with 
rounded sides (the curved glass 
serves as a lens). Place a fresh blade 
of grass in the jar. Let children de- 
scribe how the animal eats the grass^^ 

2. If it is possible to secure a ham- 
ster or white rat, have the children 
observe the kinds of food the animal 
eats, the shape of its teeth, and the 
way it eats. 

3. Have the children keep a record 
of the food they eat in one day. 
Emphasize that everything they eat. 
even meat and milk, can be traced 
back to some green plant. Point out 
the dependency of all animals on 
green plants for a food supply. 

4. Have a first-grader who has lost 
his two top front teeth describe the 
difficulty in eating such foods as 
corn on the cob and apples. Relate 
this difficulty to the relationship 
between food habits and tooth 
structure. Bring in pictures of the 
teeth of herbivores (sharp, broad, 
cutting teeth) and of carnivores 
(sharp, pointed, tearing teeth), and 
compare them. 

5. Have children discuss the foods 
their pets eat. Compare the eating 
habits and body structures of 
various types of pets. 
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FIND OUT! 

WHAT IS NEEDED 
FOR A FIRE TO BURN? 

Your teacher may light 
two candles for you. 

Put a little jar over 
one candle. Put a larger jar 
over the other candle. 

What happens? Why? 

[he flames of both candles go out, but the candle^der the 
srnollei/t^K Is ext>d^shed fir* 



.imiON: 



P.AftC 



buriTtna^o take place. 
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FIND OUT! 

HOW DO YOU KNOW THAT 
HEAT COMES FROM RUSTING? 

t 

Put a piece of steel wool 
and a little water in a jar. 

Put in a l^mometeh 
See wherelRgTrqOidin the tube 
is at first. 

After a while, look at the 
thermometer again. Did the liquid 
in the tube go up? Why? 

G^ERAl J'?^ATinM<''whVp^^ 
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THINKING AHEAD IN SCIENCE 

NOTE: See guide for responses ond suggestions. 

1. What kinds of things cannot 
be used as fuel for fire? 

2. What is a fireproof building? 

3. Where does the gas used 

in a kitchen stove come from? 
What can you do to keep things 
from rusting? 

5. What liquids are used 
in thermometers? 



THINGS TO DO 



NOTE: See guide for responses and suggcstionse 

1. Find out different ways 
of putting out fires. 

2. Find out how to keep safe 
from fire in your home. 

•3. Find out how a fire extinguisher works. 
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How do people use fire? 
They may use it for cooking. 
They may use it to keep warm. 

In what ways is fire used 
in these pictures? 
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Many times, people do not know 
' how to use fire. Then fire burns 
things that should not burn. 

People are often hurt by fire, too. 

SUGGESTION: Hove children discuss dangers involved in using fire and safeguards that hove to be 



followed. 
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The picture shows that someone suggestion; discuss vori- 

^ ous ways that forest fires may 

did not know how to use fire. start and how they con be 
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What happened? 



T7 



prevented. If some children 
have had cnnping experiences, 
have them describe how camp- 
fires are put out. 
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iswer to Question 

do these trees grow only part of 
^ year? The temperature is not high 
bugh during the winter (not enough 
\ax radiation). 
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Discussion Questions 

1. Have you ever visited a green- 
house? What was it like inside? 
Answers will vary, but should indi- 
cate the conditions of moisture, 
warmth, and brightness, and prob- 
ably the characteristic odors of 
growing plants. 

2. How is this man caring for the 
plants in the greenhouse? He is 
watering them. 

3. In what other ways may he care 
for these plants? Among others, by 
keeping the greenhouse warm (note 
the chimney from a small stove in 

• the background); by fertilizing the 
soil; by cutting and transplanting 
the plants when they grow larger. 

4. Why do florists grow plants in 
greenhouses? Plants can be grown 
almost all year round in a green- 
house, since the "weather” inside 
is controlled. Also, plants can be 
protected from insects and disease 
more easily inside than outside. 

5. Does your mother have plants 
growing inside your home? How 
does she take care of them? She 
probably waters them, places them 
in a sunny spot, keeps them out of 
cold drafts but where it does not be- 
come too hot, fertilizes the soil, etc. 



Special Activities 

1. Obtain s-ome old seed catalogues. 

1. et the children cut out pictures of 
the many kinds of flowering plants 
to apprisQ them of the variety of 
angiosperms. 

2. If there is a beekeeper in the 
community, ask him to tell the chil- 
dren about his activities, stressing 
the function of bees in pollinating 
flowers and fruit trees. Have him 
describe how bees and other insects 
and birds are attracted by the bright 
colors and odors of the flowers. 

3. Plan a visit to a greenhouse to 
note the care given to plants and 
to observe the varieties of plants 
grown. This activity is especially 
interesting in winter or early spring 



frees on the outside are not growing. 

do these trees grow only part of the year? 
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4. Provide each child with a small 
flowerpot and a small flowering 
plant, such as a marigold. Have him 
care for the plant. Discuss the kinds 
of care required. 
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the air that is needed for most burning, but 
in most orinary cases the oxygen for burn- 
ing is obtained from the air. 

€ The substance shown burning on page 81 - 
is “Sterno.” Stemo is often used as a source 
of heat on camping trips. It can be obtained 
at camping supply stores, at war surplus 
stores, and at drugstores. A candle burning 
in a bowl can be substituted for the Sterno 
in the activity illustrated on page 81. 

The pictures on page 81 show what hap- 
pens, when a fire is cut oflF from its source of 
air. in a variation of this activity, have the 
children try to predict how far over the bowl 
the cover can be pushed without making the 
flame go out After obtaining the predictions, 

try it and see. 

© One of the ways to make fuel bum faster 
is to increase the amount of air around the 
flame. We sometimes fan glowing coals in 
order to make them flare up into flames. The 
boy scout trying to start a campfire may 
blow on glowing embers in order to make 
die fire bum faster. Sometimes, bellows are 
used to start a fire in a fireplace. The black- 
smith used a bellows to make the coals in his 
forge glow when he wanted a hot fire to heat 
his m^als. In all of these cases, a fire is made 
to bum faster by supplying it with more air. 

When substances bum, they are combin- 
mg with oxygen in the air. Burning is a fast 
oxidation in which heat and light are given 
off. Fire is an e.xample of a fast chemical 

reaction. 



up the glass tubing. If a match is held next 
to the upper end of the glass tubing, the gas 
will ignite, and a small flame will form at 
the upper end of the tubing. 



C Children can learn a great deal about 
burning by observing a flame of a burning 
candle or a gas burner such as a Bunsen 
burner. The burning in such a flame actually 
I takes place at the edge '.f the flame. Children 
lean see a dark region in the center of the 
flame. This region is filled with gas from the 
candle. This gas doesn’t burn because it 
doesn’t have access to a supply of air. If the 
lower end of a piece of glass tubing is held 
,in this dark region, some of the gas will move 
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Procedure (pp. 84*85) 

Everyone, including children, should learn 
how to put out a fire. In general, fires are 
put out by depriving the flame of something 
that is needed for burning. For example, a 
candle flame can be e.xtinguished by putting 
something over it to deprive it of oxygen, as 
is shown on page 84. 

O You may wish to discuss with the children 
how they could put out fires by depriving 
the fire of air. For example, if someone's 
clothes should happen to catch on fire, one of 
the best ways to stop such a fire is to quickly 
wrap a blanket around him. The fire goes 
out because the supply of air is cut off. Some 
high school chemistry laboratories have 
blanket rolls that are readily available if a 
sleeve or other garment should catch on fire. 

If there is such a device available in tlie 
community, you may want to take the chil- 
dren to see it. 

0 Another way to remove the supply of air 
from a flame is to cover it with soil or sand. 
This is a common way to e.xtinguish camp- 
fires. It is also a way to put out small grass 

■ fires that sometimes start during a dry period. 
Make certain that the children understand 
the scientific principle underlying this 
method; the fire is being deprived of one of 

■ the requirements for burning, 
e Try to find out how the school building is 
protected from fire. Many buildings have 
automatic sprinklers that spray water 
throughout the room if a fire should start. 
Usually, there are fire extinguishers available 
in strategic places. Many school buildings 
have steel fire doors, in corridors, that are 
designed to prevent fires from spreading 
throughout a building. You can consult the 
school custodian or plant manager to get 
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energy of the atoms or molecules that make 
up a substance. However, most substances 
expand when they are heated and contract 
when they are cooled. Tlie licjuid in a ther* 
mometer, for example, e.xpands when it is 
heated and contracts when it is cooled. We 
can use this property of such a substance to 
measure the temperature. 

It is well to review with the youngsters the 
concepts of hotness and coldness. The pic- 
tures on page 90 show places that have 
radically diflFerent temperatures. 

O On page 91 are shown various kinds of 
thermometers that are used to measure tem- 
peratures. Obtain as many of these different 
kinds of thermometers as possible, and show 
them to the children. Have them put their 
fingers on the bulb of the ordinal air ther- 
mometer and watch the liquid in the 
thermometer rise. Have them note that the 
liquid “falls” when they take their fingers 
away. 






0 This is a good time to give children prac- 
tice in reading thennometers. Draw a large 
thermometer on a piece of tagboard. Indicate 
numbers at inter\'als of 10, from 0 to 120, 
along the side of the sketch so they resemble 
numbers on a real thermometer. Use a red 
ribbon or piece of paper to simulate the 
liquid in a thermometer. Move the ribbon up 
and down on the tagboard thermometer, and 
have the children call off the temperatures 
that are indicated. Following this, give their 
practice in reading real thermometers. 

O Discuss with the children various situa- 
tions in which it is important to know what 
the temperature is and how to control it. Of 
course, it is important to control the tem- 
peratures in our homes and classrooms. On 
page 92 the picture shows how the tempera- 
tures in an oven are measured. Ask the chil- 
dren why it is important to have the right 
temperature when food is being cooked in 
the oven. 
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FIND OUT! 

(p. 93) 

HOW DOES A THERMOMETER SHOW 
CHANGE IN TEMPERATURE? 

. Materials Needed: 
thermometer 
bowl 
ice cubes 

' Have the children notice the direction in 
which the liquid in the thermometer moves 
when it is cooled and when it is heated. 
What is the lowest temperature indicated by 
the thermometer when it is placed in the ice? 
What is the highest temperature that is indi- 
cated after the ice melts? 



FIND OUT! 

(p. 93) 

HOW DOES THE TEMPERATURE 
CHANGE FROM DAY TO DAY? 

Materials Needed: 
outdoor thermometer 
chart paper 

This is an e.xcellent activity for giving chil- 
dren practice in reading thermometers as 
well as for helping them to become aware of 
how temperatures may change from day to 
day. 

In order to make the temperature record 
of some value, it is desirable to observe the 
temperature at about the same time each 
day. Each child should have experience in 
reading the thermometer. 

Key to 

LOOKING BACK 
(p. 94) 

1. Tlie temperature rises when things get 
hotter. 

2. The temperature becomes lower when 
things get colder. 
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wer to Question 

I do these trees grow only part of 
year? The temperature is not high 
iigh during the winter (not enough 
ir radiation). 





Discussion Questions 

1. Have you ever visited a green- 
house? What was it like inside? 
Answers will vary, but should indi- 
cate the conditions of moisture, 
warmth, and brightness, and prob- 
ably the characteristic odors of 
growing plants. 

2. How is this man caring for the 
plants in the greenhouse? He is 
watering them. 

3. In what other ways may he care 
for these plants? Among others, by 
keeping the greenhouse warm (note 
the chimney from a small stove in 
the background); by fertilizing the 
soil; by cutting and transplanting 
the plants when they grow larger. 

4. Why do florists grow plants in 
greenhouses? Plants can be grown 
almost all year round in a green- 
house, since the “weather” inside 
is controlled. Also, plants can be 
protected from insects and disease 
more easily inside than outside. 

5. Does your mother have plants 
growing inside your home? How 
does she take care of them? She 
probably waters them, places them 
in a sunny spot, keeps them out of 
cold drafts but where it does not be- 
come too hot, fertilizes the soil, etc. 
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fees on the outside are not growing, 
hy do these trees grow only part of the year? 
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Special Activities 
1 . Obtain some old seed catalogues. 



Let the children cut out pictures of 
the many kinds of flowering plants 
to apprise them of the variety of 
angiosoerms 



2. If there is a beekeeper in the 
community, ask him to tell the chil- 
dren about his activities, stressing 
the function of bees in pollinating 
flowers and fruit trees. Have him 
describe how bees and other insects 
and birds are attracted by the bright 
pnlnrs and odors of the flowers. 




3. Plan a visit to a greenhouse to 
note the care given to plants and 
to observe the varieties of plants 
grown. This activity is especially 
interesting in winter or early spring. 

4. Provide each child with a small 
flowerpot and a small flowering 
plant, such as a marigold. Have him 
care for the plant. Discuss the kinds 
of care required. 
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IND OUT! 




IW DOES A THERMOMETER 
low CHANGE IN TEMPERATURE? 


•O. 




Put some ice cubes in a bowl 
ith a thermometer. See where 
le liquid in the glass tube is. 

Now let all the ice melt, 
nd wait a while. 
i the liquid in the tube 
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ENERALIZATION: When the liquid in a thermometer is heoted) it expands and rises. When it is 
►led, it contracts and moves down in the tubge— 



•|ND OUT! 

OW DOES THE TEMPERATURE 
HANGE FROM DAY TO DAY? 

I 

I 

I 

I 

Read an outside thermometer 
it the same time each day. 

Vrite down the temperature 
lach day for a week. 

I When was the temperature 
j ^igher? When was it lower? 
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Impressively 

Outstandingly 

Strikingly 

Skillfully 

Cleverly 



2, Clearly 
fSompletely 
Effectively 
Sufficiently 
Interestingly 
Appropriately 



3« Adequately 
Suitably 
Satisfactorily 



4. Routinely 
Commonly 
Unclearly 
Incompletely 
Ineffectively 
Insufficiently 
Inappropriately 



Erroneously 
Poorly 
Weakly 

Stercotypcdly 
Boringly 
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page of Science Given by Assignable Unit **Science is:” 



[ttacfc'lng Problems 



Classifying Thing s 



C ooperative Human Endeavor 



^eative Human Endeavor 



Curiosit y 






Disc overing M ysterious Phenomena 



^ pioring 



Fa cts 

i^eas 



Learning about >fyself 
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Verifying Things 
Working Carefully 
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Appendix Ot li^lciiic "OrmSt 



OUTLINE OF EPIE INTERVIEW SCHEDULE FOR ELEMENTARY SCIENCE TEACHERS 

INTERVIEW SCHEDULE: R 3 

Not for distribution. EPIE, 1967 



I. Introduction 



A. Overview 

B. Purposes 

C. Agreement 

II. First lud^ent of science materf ^ls 

/ 

A. Identification 

B. Reaction 

C. Describing materials 

III. Intentions - outcomes 

A. Aims for learners 

B. Outcomes observed by teacher 

C. Authors* alms 

D. Comparisons 

IV. Conditions of use It Antecedents 

A. Learners 

B. Classroom 

C. Teacher 

D. Administration - supervision 

E. Community 

V. Conditions of use II: Transactions 

A - N Instructional organization, style 

VI. Final judgment 

A. Overall 

B. Reservations 

C. Strengths 

VII. Closing procedures 

A. Quotes 

B. Setting 

C . Respondent ’ s background . 



Note to interviewer: All sentences written in capital letters 

in quotations arv? to be read aloud. As quotable comnents 
during the course of the interview, please jot down as 
much as possible in Section VII: Summary Procedure, or in mar 

gins when they occur. 

.236 . 
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2 . 



I. INTRODUCTIO N 

I 

A, Overview! (Exchange pleasantries. Do not discuss EPIE or 
interview.) 



"MISS , I ASSUME WE»D AGREE THAT YOUR DECISIONS 

ABOUT SCIENCE MATERIALS COULD BE IMPROVED BY YOUR HAVING INFOR- 
MATION ON HOW WELL THEY WORKED FOR OTHER TEACHERS? (Pause for 
assent.) GOOD, NOW I WOULD LIKE TO READ A STATEMENT VJHICH DES- 
CRIBES THE PURPOSES FOR TODAY’S INTERVIEW. AFTER YOU HEAR ^HE 
STATEMENT YOU MAY HAVE SOME QUESTIONS. I’LL BE HAPPY TO AN- 
SWER THEM. 



"EPIE STANDS FOR EDUCATIONAL PRODUCTS INFORMATION EXCHANGE, A 
NONPROFIT AGENCY CREATED BY PROFESSIONALS FROM ALL QUARTERS OF 
THE EDUCATIONAL COMMUNITY. EPIE GATHERS INFORMATION ABOUT THE 
ACTUAL PERFORMANCE OF SCHOOL MATERIALS. RIGHT NOW, IT IS WORK- 
ING ON ELEMENTARY SCIENCE MATERIALS. 



"EPIE WANTS TO ENCOURAGE INNOVATION 01 NEW EDUCATIONAL MATERIAL? 
AND Tl^ROVSMENT IN THE DESIGN AND USE OF _XISTING MATERIALS. EPIE 
WILL MAKE ITS INFORMATION AVAILABLE TO EDUCATORS LIKE YOU TO HELP 
SELECT WISELY AMONG THE INCREASINGLY NUMEROUS AND COMPLEX EDUCA- 
TIONAL MATERIALS BECOMING AVAILABLE. AFTER WE COMPLETE OUR INFOR- 
MATION GATHERI1C3, YOU CAN FIND OUT IN A PUBLICATION CALLED EPIE 
FORUM HOW IT IS AVAILABLE TO YOU AND OTHER TEACHERS.." 



B. Interview Purposes 



"DURING THIS INTERVIEW, I WILL BE ASKING YOU A VARIETY OF QUES- 
TIONS ABOUT YOUR EXPERIENCES USING SCIENCE MATERIALS FOR CLASS- 
ROOM INSTRUCTION. EPIE DOES NOT KNOW IF THE QUESTIONS IT ASKS 
ARE ALL THE RIGHT ONES OR IF THEY ARE ALL NECESSARY. I HOPE 
YOU WILL BE TOLERANT IF A FEW QUESTIONS SEEM OF LITTLE VALUE 
TO YOU. IF WE ERR IN OUR QUESTIONING, WE HOPE IT IS IN THE 
DIRECTION OF OVER- DESCRIBING THE INFORMATION YOU HAVE ABOUT 
SCIENCE MATERIALS." 



C. Agreement 



"NOW, YOUR SPECIFIC ANSWERS TO THESE QUESTIONS WILL BE SHARED 
WITH NO ONE OUTSIDE EPIE WITHOUT YOUP CONSENT. WE ARE RETAIN- 
ING YOUR NAME TO PUT THIS INFORMATION TOGETHER WITH INFORMATION 
YOU MAY GIVE US IN THE FUTURE. THE' INTERVIEW USUALLY TAKES 40 
MINUTES. IF YOU DON’T UNDERSTAND A QUESTION, PLEASE LET ME 
KNOW." 
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3. 



II. FIRST PROBE FOR JUDGMENT OF SCIENCE MATERIALS L 
A* Product Identification! 

"WHAT TEXTBOOK AND OTHER INSTRUQTIONAL MATERIALS ARE YOU USING IN 
YOUR SCIENCE TFACHING THIS YEAR?" (If teacher mentions using two 
or more texts for science Instruction, say) "WHICH OF THESE DO 
YOU CONSIDER YOUR PRIMARY TEXT? THEN LET’S TALK ABOUT THAT ONE 

tl 



Name of text and series 



Other materials 






B. First Judgmental Reaction; 

1. "PLEASE GIVE ME A GENERAL REACTION T O (name of textbook) ." 
(Whatever teacher says should be followed by your saying,) 

"TELL ME MORE ABOUT . " (the principal 

theme of the teacher’s general reaction „ Interviewer follows 
the leadr of the teacher without any attempt to guide, nor to 
explore areas not mentioned by the teacher*. Use the back of 
facing page to note the teacher’s general reaction.) 

2. "WHAT IS THE STRONGEST FEATURE OF THE TEXT?" (Check any that 
are mentioned. Do not read the list or suggest.) 

student affect! challenge, enthusiasm, interest, moti- 
vation 

aid to teacher: demonstrations, tests, teaching tips 
problems, projects, experiments for students 
physical features: durability, size* print 
diagrams* graphs, charts, maps, pictures 
readability, clear, common pense approach 
science content covered, ordering of topics 



other 




o 
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4. 



3. "WllAT IS THE GREATEST FAULT?" 

(Check each as mentioned. Do not read the list or suggest. 
If any of the first three * items are mentioned, then you 
ank the teacher:) 

"WHAT DO YOU DO TO STRENGTHEN THIS ASPECT OF THE TEXT?" 



* b. Qj poor coverage of science content; or ordering of topics 



'* c. [^! inadequate problems, projects, experiments for pupils 



d. student affect: challenge, enthusiasm, interest, 

motivation 

e. Q physical features bothersome: durability, print size, 

booksize 

f. Qj readability level inappropriate: lacks clarity 

g. Q insufficient mraiber or poor quality of charts, dia- 

grams, pictures, drawings 



4. "HAVE YOU COMPARED lUIS TEXT WITH ANOTHER COMPETING TEXT 
RECENTLY?" 




needs more help for teacher: 

tests, evaluation procedures, demonstrations 



Remedy : 



Remedy : 



Remedy ; 




other 



□ no 

□ 



name 



(If 'yes':) "WHAT DIFFERENCES DID YOU FIND?" 



ERIC 
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5. 



5. "IF YOU HAD AN OPPORTUNITY TO SPEAK TO THE PRODUCERS OF 
ALL ELEMENTARY SCIENCE TEXTS AND MATERIALS, WHAT WOULD 
THE MAIN IDEA OF YOUR MESSAGE BE?" 




change 




add 




delete 




no message 



6. (Interviewer makes a judgment indicating the teacher’s 
overall satisfaction with the text* Do not read*) 



strong 

ambivalent 

feeling 















« 






highly 


dissatis- 




satisfied 


highly 



dissatis- fled with with some satisfied 

fled some sup- little reserva- 



port apparent tlons 

feeling 
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II. C. ’’LET’S TURH now TO A FEW SPECIFIC QUESTIONS ABOUT THE MA'IERIALS. 
FIRST, THE TEXTBOOK: HOW DO YOU FIND ITS READ^MLOT?” (Read 

the list.) 







Mixed No 

Feel- Opln- 

Ings Ion 






TEXTBOOK 










a. READABILITY 


Easy 


□ □ □ □ 


Difficult 


b. CHOICE OF SCIENCE 
CONTENT: TOPICS 


Appro- I 
prlate | 


n □ □ 1 


11 


Inappro- 

priate 


c. ORDER OF CONTENT 


Appro- j 
prlate 1 


□ □ □ 1 


□ 


Inappro- 

priate 


d. DURABILITY 


Wears I 
well ' 


□ □ □, 1 


□ 


Wears out 


e. ILLUSTRATIONS 


Good 


□ □ □ 1 


□ 


Poor 


EXERCISES & PROBLEMS, 


TEXT OR WORKBOOK 






a. CHOICE 


Rele- 

vant 


□ □ □ 


□ 


Not relevant 


b. DIFFICULTY LEVEL 


Appro- 

priate 


□ □ □ 


□ 


Inappro- 

priate. 



3. GUIDE TO TEACHER. TEACHER’S EDITION (Including preface and supple- 
mentary materials) 



PHILOSOPHY, 
STATEMENT OF 
GOALS 


Well 1 

Stated 1 


□ 


□ 


□ 


□ 


Not apparent 


SUGGESTIONS FOR 
ACTIVITIES 


Help- I 
ful 1 


□ 


□ 


1 □ 


□ 


Not helpful 


EVALUATION SUGGES- 
TIONS AND TESTS 


Useful 1 


□ 


D 


1 □ 


□ 


1 Not useful 


ADDITIONAL READ- 
INGS, FILMS & 


Useful 


□ 


C 


1 □ 


c 


1 Not useful 



FILMSTRIPS 
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4 . 



Ask the teacher, "CONSIDMING EVERXHTNG WE HAVE 
TO THIS POINT, WHAT IS YOUR OVERALL JUDGMENT OF 
ITY OF THE MATERIALS?” 



DISCUSSED 
THE QUAL- 



strc. « 

ambivalent 

feeling 











1 

1 












i 

1 












highly 

dissatis- 

fied 






satisfied 
with some 
reserva- 
tions 


highly 

satisfied 


Q.LS ScluXS 
fled with 
some sup- 
port 


little 

apparent 

feeling 
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III. ^ iSTXONS ON INTENTIONS 



A. (Repeat the following stem for each of the 5 alms.) • "HOW MUCH 
EMPHASIS DO YOU PLACE ON HAVING YOUR STfJDENTS:" 



Teacher 

doesn't 

know 



(A) 



1. LEARN WHAT THE MAJOR BRAN- 
CHES OF SCIENCE ARE (BOT- 
ANY, PHYSICS, SOCIAL SCI- 
ENCE,...)? 




2. UNDERSTAND HOW SCIENTI- 
FIC RESEARCH HELPS EX- 
PLAIN THE PROBLEMS OF 
DAILY LIVING? 




3. ACQUIRE A GENERAL APPRE- 
CIATION FOR ALL FORMS OF 
SCIENCE? 




4. THINK AND ACT INDEPEN- 
DENTLY USING SCIENTI- 
FIC METHODS OF INQUIRY? 




5. LEARN WHAT IS NECESSARY 
FOR LATER SCIENCE COURSE- 
WORK? 




No Minor 

empha~ empha- 
sis sis 

(B) (C) 

□ □ 

□ □ 

□ □ 

□ □ 

□ □ 



Aver- Great 
age empha- 

eropha- sis 
sis 

(D) (E) 

□ □ 

□ □ 

□ □ 

□ □ 

□ □ 



B. "NOW LET'S CONSIDER YOUR EMPHASIS AGAIN. DO YOU FEEL THE AIMS YOU 
HAVE JUST IDENTIFIED ARE BEING ACHIEVED IN YOUR SCIENCE INSTRUCTION?" 
(^ead only those aims which th(si teacher said received average or 
great emphasis in III A.) 



Emphasis 1 
2 




3 



9o 



C "AUTHORS OF COURSE, HAVE OBJECTIVES TOO. LET’S GO THROUGH THESE AIMS 
AGAIN FOR STILL ANOTHER REASON; TO RECORD WHAT YQU SEE YOUR TEXTBOOK S 
AUTHOR’S COALS TO BE. THINK BACK TO WHAT IT SAYS M ^ 

FACE, AND WHAT IS EMPHASIZED IN THE TEXT. (Pause.) NOW, HOW MUCH 
EMPHASIS DO TliE AUTHORS PLACE ON HAVING YOUR STUDENTS; 



1. LEARN WHAT THE MAJOR BRAN- 
CHES OF SCIENCE ARE (BOT- 
ANY, PHYSICS, SOCIAL SCI- 
ENCE,...)? 

2. UNDERSTAND HOW SCIENTI- 
FIC RESEARCH HELPS EX- 
PLAIN THE PROBLEMS OF 
DAILY LIVING? 

3. ACQUIRE A GENERAL APPRE- 
CIATION FOR ALL FORMS OF 
SCIENCE? 

A. THINK AND ACT INDEPEN- 
DENTLY USING SCIENTI- 
FIC METHODS OF INQUIRY? 

5. LEARN WHAT IS NECESSARY 
FOR LATER SCIENCE COURSE- 
WORK? 



Teacher No 
doesn’t empha- 
know sis 

(A) (B) 


Minor Aver- 

empha- age 
sis empha- 

sis - 
(C) (D) 


Great 

empha- 

sis 

(E) 


□ 


□ 


□ 


□ 


1 i 


□ 


□ 


□ 


n 


□ 


□ 


1 □ 


□ 


1 n 


n 

1 .1 


c 


1 □ 


□ 


1 n 


I : 

j_i 


c 


] □ 


r" 

t 

I 


1 n 


r"i 

i j 



D "DO YOD FEEL THERE ARE DIFFERENT AIMS TO BE FOUND BETWEEN THE MTERIALS 

o“diffeS publishers ... or are they pretty much all the same?" 

same [J differences exist 



j~ "j no opinion the 



(If the teacher feels there are differences, ask;) ^ 

PUBLISHER AND SERIES’ NAME AND THE DIFFERENCES IN THEIR AIMS. 

Author’s Emphasis 

Aim(s) Given 



1 . 



Series or publisher 



2 . 



(Note to interviewer: use the numbers 1-5 (see C above) ss a 

the author's aim, and the letters A-E (see C above, also) as a code for 

the emphasis given.) 
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IV. SPECIFICATIONS FOR ANTECEDENTS 
A. Learners 

1. "WHAT IS THE AVERAGE AGE OF YOL^ PUPILS?” 



2. "NOW THINK ONLY OF CHILDREN OF AGE . CONSIDER THEIR 

INTELLIGENCE. WOULD YOU SAY THAT THESE MATERIALS (name of 
materials) ARE APPROPRIATE FOR ALL LEVELS OF I.Q.? 



3. 



"FOR LOW?" 


"FOR MIDDLE?" 




"FOR HIGH?" 


C y®" 


□ y®® 




□ 


yes 


□ »o 


□ 




□ 


1 no 


1 ] dnk 

(do not 
know) 


1 ~1 dnk 




11 


1 dnk 


(Use for Grades 2-8 only) 

"ARE THESE MATERIALS APPROPRIATE FOR 


STUDENTS WHO READ 


ONE YEAR BELOW GRADE LEVEL?" 


□ 


yes F 


1 [ 



TWO YEARS BELOW GRADE LEVEL?" Q y®® Q [3 

THREE YEARS BELOW GRADE LEVEL Q] F} U 

4. "ARE THESE MATERIALS EQUALLY APPROPRIATE FOR BOYS AOT GIRLS?" 

ri more for F] equal H more for Q dnk 

LJ Boyg L-J * Girls 



5. "ARE THESE MATERIALS FOR PUPILS 
OF HIGH MOTIVATION?" 

OF LOW MOTIVATION?" 



[J yes [J Q 

Fj y®® FI FJ 



6. "DO YOUR PUPILS GENERALLY LIKE THESE MATERIALS?" 

Q yes [_] no Q 
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B. Classroom 



1. "CAN YOU THINK OF ANY SPECIAL EQUIPMENT THAT YOU HAVE TO HAVE 
BECAUSE YOU ARE USING (these materials) THAT YOU WOULDN’T HAVE 
TO HAVE USING OTHER SCIENCE HATERIALS?" 



L y®® ^ THEY?" 

r- 

no 



2. "CAN YOU THINK OF ANY C LASSROOM ARRANGEMENTS. THAT YOU HAVE TO 
HAVE BECAUSE YOU ARE USING THESE MATERIALS THAT YOU WOULDN’T 
HAVE TO HAVE USING OTHER SCIENCE MATERIALS?" 



yes "WHAT ARE THEY?" 
. j no 



3. "CAN YOU THINK OF AN Y ADMINISTRATIVE CLEARANCE THAT YOU HAVE 
TO HAVE BECAUSE YOU ARE USING THESE MATERIALS THAT YOU WOULDN’T 
HAVE TO HAVE IF YOU WERE USING OTHER SCIENCE MATERIALS?" 





yes 

no 



"WHAT CLEARANCE?" 




1.36 
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C. Teacher 



1. "CONCENTRATE ONCE AGAIN ON YOURSELF IN THE CLASSROOM DURING 
YOUR SCIENCE TEACHING. DO YOU SEE YOURSELF AS MORE ON THE 
PRACTICAL SIDE OR AS MORE ON THE IDEALISTIC SIDE?" (Pause, 
now choose one of the following to read, based on the teach- 
er’s reply.) 



(If "practical", read:) ^ 

"WOULD YOU SAY YOUR TEACHING IS DOWN-TO-EARTH, TOWARD PRAC- 
TICAL, USEITJL THINGS RATHER THAN IDEALISTIC, TOWARD ABSTRACT, 
LONG RANGE VALUES?" 



^If "Idealistic", read:) 

"WOULD YOU SAY YOUR TEACHING EMPHASIZES THE IMPORTANT LONG 
RANGE HUMAN VALUES RATHER THAN PRACTICAL, DOWN-TO-EARTH, 
DAY-TO-DAY PROBLEMS?" 



(If "undecided", read:) 

"DO YOU FEEL THAT YOUR TEACHING IS ABOUT EQUALLY CONCERNED 
WITH IMPORTANT, LONG RANGE HUMAN VALUES AND PRACTICAL, 
DOWN-TO-EARTH, DAY-TO-DAY PROBLEMS?" 



Practical Practical Undecided 
But 

□ □ □ 



Idealistic 

But 




Idealistic 

□ 



2. "HERE IS ANOTHER DIFFICULT CHOICE FOR MANY TEACHERS: DO YOU 

CONSIDER YOURSELF MORE ON THE FLEXIBLE SIDE OR MORE ON THE 
ORDERLY SIDE? THAT IS, DO YOU LIKE - MORE OR LESS - TO LET 
THE THINGS THAT HAPPEN MAKE THE SCHEDULE, OR DO YOU LIKE TO 
HAVE A FIRM SCHECULE AND SEE THAT THINGS HAPPEN ACCORDINGLY? 



(If "flexible" f read:) 

'WOULD YOU SAY THAT YOU ARE MORE FLEXIBLE THAN - SAY - A VAST 
MAJORITY OF THE TEACHERS YOU KNOW?" 



It 



(If "orderly", read:) 

"WOULD YOU SAY THAT YOU ARE MORE ORDERLY, BETTER ORGANIZED 
THAN - SAY - A VAST MAJORITY OF THE TEACHERS YOU KNOW?" 



(If "undecided", read:) mvAnu 
"DO YOU FEEL THAT YOU ARE PRETTY MUCH THE SAME AS OTHER TEACH- 
ERS AS FAR AS BEING FLEXIBLE AND ORDERI^Y IN THE CLASSROOM? 

Flexible Flexible Undecided Orderly Orderly 
But 



n 



□ u 
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"NOW WOULD TOU SAY THAT YOU ARE RELATIVELY HAPPY OR RELATIVE- 
LY DISSATISFIED WITH THE TEACHING JOB THAT YOU HAVE? JUST HOW 

DO YOU FEEL?" 



Quite Rather So-So 

pleased satisfied 



Somewhat Quite 
dissatis- dissatisfied 
fied 




"YOU HAVE GIVEN US SOME INSIGHT INTO HOW YOU FEEL ABOUT YOUR 
WORK. HOW DOES YOUR PRINCIPAL VIEW YOU?" (Do not read or 

suggest.) 



Positive 

Strong 




Negative 



omewhat 


Somewhat 


Strong 


I [ (Support) 


n ' 


□ 


; 1 (Personal feeling) 


□ 


n 



"WILL YOU BE TEACHING HERE OR ARE YOU CHANGING POSITIONS 
NEXT YEAR?" 




j j undecided 



(If "no" ask;) "WOULD YOU CARE TO TELL ME WHY YOU ARE CHANG- 
ING YOUR POSITION?" 




to take a position in this same school more appro- 
priate to training 




circumstances dictate it; school closing, getting 
married, pregnant, etc. 

seeking better teaching assignment 
other 

no comment 



14. 



D» Administration 



i. "WOULD YOU DESCRIBE YOUR PRINCIPAL MORE AS AN ADMINISTRATION 
SPECIALIST OR AS A CURRICULUM COORDINATOR?" 



□ 

□ 

□ 



administration specialist 

curriculum coordinator 

neither, both, do not know, profanity 



2. "DO YOU HAVE AN ELEMENTARY SCIENCE COORDINATOR?" 



I I yes 




no 



I I dnk 



3. (If yes) "HOW FREQUENTLY DO YOU 
AND THE SCIENCE COORDINATOR DIS- 
CUSS SCIENCE INSTRUCTION?" 



(If no) "HOW FREQUENTLY DO 
YOU AND THE PRINCIPAL DIS- 
CUSS SCIENCE INSTRUCTION?" 



□ 

□ 

□ 

□ 

□ 



Almost every day 

2-3 times per week 

2-3 times per month 

sometimes, but less than once a month 

almost never 



4. "TO WHOM WOULD YOU ’WRITE' IF YOU WANTED TO MAKE A STRONG 
PETITION AGAINST ANY POOR INSTRUCTIONAL MATERIALS YOU WERE 
EXPECTED TO USE?" 



[ j supervisor 


□ 


J principal 


□ 


1 1 superintendent 


□ 


[ [ school board 





(Interviewer may substitute "call" 



public figure outside school 

other 

I would not make such a petition 



or "speak" for "write.") 
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5. "WHO SELECTED YOUR TEXT FOR YOUR CL^SROOM USE?" 



r~! 

LJ 

□ 

□ 

□ 

□ 

□ 



curriculum committee 
science supervisor or his staff 
principal or superintendent 
teacher 

teachers In the building 
other 



6. "IS THIS TEXT ADOPTED SYSTEM-WIDE?" 





yes 



other 



E. Community 

"DESCRIBE THE NEIGHBORHOOD (S) YOUR SCHOOL SERVES; 

a. "SOCIO-ECONOMIC STATUS" 

□ ■ □ □ □ 

Disadvantaged Middle Class Highly Mixed - high 

privileged contrast 

b. "PARENTS' INVOLVEMENT IN SCHOOL AFFAIRS" 

□ □ □ 

Disinterested Average High Involve- 

ment 

c. "GENERAL PUBLIC'S ATTITUDE TOWARD TEACHING OF SCIENCE" 

□ □ □ 

Like any Slightly Considered to 

other sub- more Inter- be very Impor- 

ject ested In scl- tant 

ence 



•ikO • 
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V. '.CRANSACTIONAL CONDITIONS OF USE 



A. "HOW LONG IS A TYPICAL SCIENCE CLASS PERIOD?" 

□ □ □ □ p 

15 min. 30 min. 45 min. hour + other 



B. "HOW MANY OF THOSE PERIODS DO YOU HAVE PER WEEK?" 

□ □ □ □ □ 

less 1 2 3-4 other 

than one 

C. "HOW MANY PUPILS DO YOU TEACH SCIENCE TO EACH WEEK?" 

□ □ □ .□ 

1 class combined several other 

classes classes 

(team) 

Dv "THEN YOU TEACHi THE SAME (C) PUPILS IN SCIENCE FOR 

ABOUT (A x B) MINUTES PER WEEK?" 



E. ."IN A TYPICAL SCIENCE PERIOD OF (A) MIWTES, HOW MUCH 

TIME IS SPENT STUDYING THE TEXT (AND/OR WORKBOOK)?' (You may call 
it study time, reading assignment or something like that.) 

' F. "DO YOU HAVE STUDENTS WORK ON INDIVIDUAL SCIENCE PROJECTS OR LAB- 
ORATORY EXERCISES DURING THE YEAR?" 



[ I yes Q oo n occasion 
G. "DO ALL PUPILS GET ABOUT THE SAME ASSIGNMENTS?" 



r~l yes 



r I most with 
a few ex- 
ceptions 



I I individualized 
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H. "HOW ARE THE PROJECTS SELECTED? 

1 . Q] "COMPLETELY BY EACH PUPIL? 

2. Q] "INDIVIDUALLY FROM THE TEACHER’S LIST? 

3^ [ ^ "ASSIGNED TO EACH PUPIL BY THE TEACHER?" 

I "ABOUT WHAT PERCENT OF THE TOTAL CLASS TIME DURING THE YEAR IS 



SPENT IN INDIVIDUAL AND GROUP PROJECTS? 



□ □ □ □ □ 



25 X 50 % 75 X 




THE CLASS AS A WHOLE?" 



□ 



□ 



subgroups 



whole class 



o 



18. 



K. "SOME TEACHERS SP^ “S^G^DlS^SlS^ GIVING 

THINGS WORK, HOLDING RECITATIO , 

QUIZZES. 

1. "HOW IS MOST OF YOUR CLASS TIME SPENT? 

r — 1 el ^1 LABORATORY 

Q lecturing ® U 

PIDEMOHSTRATIHG HOH f □ 

^THINGS WORK ^ q INDIVIDUAL PROJECTS 



a 

b 



‘ things work 
I I recitation 
I [discussion 



h D 



other 



2. "HHICH of these do tou spend hegi,igxble or no time on at AI.LI 



a P'l lecturing 

b I ^[demonstrating how 
’ — • things work 

C Q kecitation 

d Pi discussion 



j I laboratory 

f pj QUIZZES 
g PJ INDIVIDUAL PROJECTS 

h a 

other 



3. "DO TOO SPEND HALF-TIME OH .entloned In # T7 

1 yes 1 1 



(other) 



4. "DO YOU SPEND ABOUT AS MUCH TIEM ON another from f i7 

?'* 



AS YOU DO ON 



(the one named first) 



(Keep cutting the 

areas in # 1 with the mo s » ^ format in question # 1 

;t r^e^tiS^tif - not teea aouH 

to teacher.) ^ time spent 

% time spent 



lecturing 

demonstration 

recitation 

discussion 



laboratory 

quizzes 

individual pro- 
jects 

other 



100 X Total 



o 
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L. "HOW CLOSELY DO YOUR CLASSTIME ACTIVITIES FOLLOW THE TOPICS IN 
THE TEXTBOOK? (AND/OR WORKBOOK)?" 



□ Very I I Sone^rtiat | | Not 

closely I I closely L_J st sll 



M. "WHICH OF THESE WOULD BEST DESCRIBE THE PUPILS IN YOUR SCIENCE 
CLASS?" 

HARD WORKING AND ENTHUSIASTIC 

remarks 

P~| hard working and well disciplined 

I j BELLIGERENT AND UNRULY — 

N. "HAVE YOU CHANGED YOUR TEACHING STYLE OR TECHNIQUE SINCE YOU STARTED 
USING THESE PARTICULAR MATERIALS?" 



"IN WHAT WAY?" 



(If "yes") "WAS THE CHANGE MADE LARGELY BECAUSE OF THE REQUIREMENTS 
OF THESE MATERIALS?" 

j I yes Q 





VI. FINAL PROBE FOR JUDGMENTS; 

A. "CONSIDERING OUR TOTAL DISCUSSION TODAY AND ADDING ANYTHING I HAVE 
FAILED TO MENTION, HOW GOOD ARE THESE MATERIALS, ALL IN ALL?" 



strong 

ambivalent 

feeling 





— 






! 

1 

1 

1 

1 

1 

>• • ••• ** * 

hlffhlv 


dlssatls- 






satisfied highly 



dissatis- 

fied 



fled with 
some sup- 
port 



little 

apparent 

feeling 



with some 
reserva- 
tions 



satisfied 




- 
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B. 1. "OK, NOW FOR THE LAST TIME, PLEASE SUMMARIZE YOUR GENERAL 
RESERVATIONS ABOUT THE MATERIALS." 





2.- "PLEASE SUMMARIZE AGAIN THE STRENGTHS OF THE MATERIALS AS YOU 
SEE THEM." 






VII. SUMMARY PROCEDURE; 



Quotable commenta; (Phrases used by the teacher du g 
of the l^atervlew which seem particularly revealing of h j g® 
of the products' performance, strengths, weaknesses.) 



1 . 
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B. Interview Setting; 

1. Date 

2. Teacher 

3. Place 

A . Interviewer 

5. Where 

6. Time 

(length) 

7. Unusual events during Interview: 



8 . 



Problems related to Interpersonal dimension as It developed In 
Interview. So you have any reason to feel that the ^&ta from 
cMs inteiTJ’iew should he held suspect? 



di no d! yoF 
(If “yes" indicate part(s) of interview concerned; why.) 



T 
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VII. C. EPTE INTERVIEW! TNFQBMATION SHEET 



Interviewer 
I. BAck^round Data!, 



Month;, Dayp Year 



Last name 



First 



School 



Grade 



Teaching responsibility 



Years 

taught 



Years with materials 



Highest degree 



Date Major 
con- Field 
ferred 



Who selected materials 
for use? 



How long (yrs.) iu How long 
use now? will they 

be in use? 



II, SinmnAr Y Reaction t 



er|c 



Poor With Am- 
or Reser- biva- 
worse vations lent 



Judgment of product \ ■ ^ 1 1 I — 1 



Positive Good 
reaction or 
in gen. better 

n □ 



□ 

□ 



□ 

□ 



□ 

□ 



L-J 

□ 

□ 



□ 

□ 



□ □ □ □ □ 



Judgment of self as 
teacher of product 

Prognosis for future 
of product in your 
class 

Your assessment of 
whether this inter- 
- view accurately 
portrays your 
judgments 

them first, of course. 



□ 



yes 



□ 



other 

• ivr " 






Appendix B: Draft Manxiscflpt 



The Anatomy of a Prototype Institute to Train 
Information Research Associates 

/ by 

Robert T, Filep 



James Boswell has written that knowledge is of two kinds: we know a 

subject ourselves, or we know where we can find information about it. . 
Chances are that when you identify a colleague as a source of information 
and inquire about his use of an educational product, you probably will 
receive either one of two kinds of answer. First, there is the type of 
statement that is brief yet loaded with emotional connotations; and 
second, there is the other type, so lengthy and diffuse that you learn 
more about the product than you care to know. 

Consider then the complexity of gathering information from many such 
users, reporting this information in a reliable fashion, and synthesizing 
the data for dissemination to any potential user who inquires of a 
central source for such infoi*mation. The EPIE (Educational Products 
Information Exchange) Institute held at Southampton College of long 
Island University fx'om’, August 13-31> '^ 9^1 > dealt with aspects of Just 
this problem. 



The prime objective of this exploration and learning effort was to test 
a prototype training package which could be used for training many 
professionals in the strategy and technique of collecting educational 
product information. Upon completion of the Institute and following field 
experience, these trained professionals would then be designated as EPIE 
Information Research Associates. Their future responsibilities would be 
to collect data for EPIE from educational product users by means of a 
"standardized" interview protocol. 




Dr. Filep is an Education Systems Scientist at System Development Corporation 
(SDC), Santa Itonica, California and a member of the graduate faculty in 
Education, UCLA. The Institute described in this article was designed by 
Dr. Filep under contract by EPIE with SDC. 
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‘ . 2 . 

The need for Information Research Associates in any product information 
exchange becomes clear when one considers a construct which shows how 
product information might be obtained by EPIE (see Figure l). Initially, 
there will be a need for direct product user contact in collecting 
information. Having a sampler of opinions of products in the field, the 
IRA^provides a full opportunity for supplementing major questions required 
by the EPIE protocol. Ambiguities can be cleared up and the background of 
Important material explored at first hand. Interpretations can be checked 
and the data enriched by using the responses to develop further questioning. 

If carried far enough, the interviewee may even be able to contribute to 
the formulation of new questions that could possibly elicit the relevant 
information in a more satisfactory way. This, of course, requires that 
the IRA give the product user a reasonably complete account of the 
purposes and problems of the product survey. Consequently, an Important 
ingredient of training rdust be imparting to the IRA the theoretical and 
operational construct of EPIE objectives. Indeed^ the IRA can increase 
the value of his feedback in direct correlation to his understanding of the 
total operation of the information exchange: and the underlying rationale 
for the methods end procedures used by EPIE. 

The development of a system for creating Information Research Associates 
was an interesting challenge. Certainly it had to be viewed in both the 
theoretical and operational dimensions as a type of continuing education 
for professionals. For all the participant-professionals involved, rusty 
skills would have to be polished, new skills and outlooks gained, and 
relevant links to current job assignments identified. 

The participants attending the prototype Institute included representatives 
from three states: New York, New Jersey, and Pennsylvania. The career 

positions these educators held were:. State Title I Evaluation Coordinators, 
County Child Study Supervisors, Area Curriculum Coordinators, College 
Curriculum Center Researchers, and Regional Educational Center Representatives. 
These people were responsible for action programs in their respective 
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agencies as veil as being decision makers at various levels of authority. 
They were .students only in the sense that new ideas and techniques were 
to he learned. They brought to the Institute a day-in, day-out action- 
doing" orientation, a wide range of prior experiences and knowledge, and 
a self concept of being professionalL ^ho were identified as "knowing 
the -answers. They expected that the experiences provided by the program 
would be relevant to whatever prior concept they had of an IRA, and 
simultaneously would be partially applicable to their current positions. 



The Institute *s faculty was composed of university and research organiza- 
tion personnel currently involved with education and information problems. 
Since the task of identifying the skills and behaviors required for the 
Information Research Associate was totally new and unique, any course of 
study had to cut across a number of established disciplines. Consequently, 
it was necessary to involve related but dissimilar disciplines in order 
that the strengths and relevancies of each could be fused in the attempt to 
develop the sacred and profane facets of this new professional designation, 

the IRA. 

The faculty represented a multi-discipline background including curriculum 
evaluation research, interview protocol techniques development, tests and 
measurement development, information sciences, and computer sciences. 

All faculty members were activel^r involved in classroom teaching prior to 

the Institute. 



In addition, guest faculty from several parts of the country were invited 
to meet with the group. In each instance, these personnel were, currently 
involved in some aspect of product information collection, synthesis, and 
analysis at the local, state, or national level. (A list of participants 
and faculty is provided at the end of the article.) 



4 
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By design, the individual's role in the prototype institute is not 
designated in. the list of participants. During the three-veek period 
the continuing Zeitgeist essentially was to he a forum for the exchange • 
of ideas and the application of an iterative approach. The attempt was 
to assure feedback to faculty at every stage of the learning program. 
Consequently, two-way communication in the spirit of dialogue between 
all concerned was the goal, as contrasted to a lecture/notetaking effort. 

The general objectives for the prototype Institute were to: 



1. Establish a forum whereby participants and faculty could 
enter into a dialogue directed toward enriching and refining 
the theoretical, mathematical, and pperational languages 
involved in product information collection and exchange. 

2. Provide a developmental field trial of a training program 
designed to prepare Information Research Associates. 

3 . Acquaint participants with the objectives and purposes of 
the Educational Products Information Exchange and those of 
the EPIE Information Research Associate (iRA). 

4. Familiarize the participants, currently involved in 
education, with the approaches and methodologies to be 
used by EPIE in collecting and disseminating information • 
about the use of educational products. 

■ 5 . Train the peurticipants in aspects of educational psychology, 
educational technology, and educational measurement relevant 
to EPIE, the IRA, and their professional assignments. 
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6 . Give initial experiences and guidance in the use of a 
developmental interview protocol which would be utilized 
(by the Institute participants functioning as IRAs) 
during a field trial of the instrument in the fall of 196T. 

7, Develop an awareness on the part of the participants 
"regarding the problems that they might face when performing 

as IRAs interviewing classroom teachers, and identify 
methods for coping with these problems. 

8 . Provide opportunities for participants to evaluate the 
place of judgment in educational research design and 

reporting. 

9, Enable Institute participants to evaluate and suggest 
revisions of ERIE techniques and procedures for training 

XRASe 

It should he clear that the preceding objectives of the Institute dealt 
with conceptual constructs as well as specific techniques. In carrying 
out the objectl^'es of the Institute, a number of teaching techniques and 
medf.a were utilized. These Included: small group discussions, video- 

taping laboratories, audiovisual programed instruction, tutorial sessions 
simulation games, large group instruction, group dynamics sessions, and 
self diagnostic examinations. Some of the approaches to teaching during 
the conduct of the Institute are outlined below. 

v<^pn-tape Recording and Cri tique Sessions. The meetings 
provided an opportunity for the participants to obtain 
practice in using interview protocols. The participants 
were recorded on video-tape while conducting the interviews 
with simulated product users; Following the taping. 
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■fchcy V6rs a.bl6 to roviow thoir porfonnancs and discuss the 
strengths and weaknesses of their interview teclmiques 
with faculty members. 

To observe one’s self and be able to be a critic of one’s 
behavior, given a set of criteria, seems to be a far more 
effective method for changing performance than having 
someone tell you how you performed and suggest methods for 
modifying your behavior. In this respect, the video-tape 
recorder proved to be a powerful teaching instrument. 



Critiques and Seminars on Protocol Instrumei ^. Participants 
were encouraged to assist with the evolution of interview 
protocols, both from the standpoint of a person who would 
be using the instrument as well as that of a product user. 
Formal critiques and seminars were devoted to this topic and 
revisions of the protocol instruments were mads based upon 
student comments prior to any further student use of the 
instruments in a simulated interview session. 

Simulation Games . A game of this nature was used to provide 
the participants with a series of "experiences in 
information transfer. The game is designed to model any 
situation in which one group is responsible for improving 
the performance of another group by providing them with 
information. This game, developed by Fred L. Goodman, was 
described in the initial issue of the EPIE FORUM. The 
information exchanged in the game concerns a simple game of 
logic which is played by those who assume the role of 
information system "users." Several teams of counselors 
compete and cooperate in an effort to improve the performance 
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of those actually playing the game# A rather wide array 
of problems are imbedded in the situations, including 
those caused by too little redundancy in the message 
exchanged as well as those which occur from failure to 
see the need for "team" play in exchanging information. 

Programed Sequences . These instructional units were 
presented via synchronized filmstrip and audio tape* 
raT ing pre-» and post—tests were provided and the 
sequences required active responding on the part of the 
students. These materials were developed by Vimcet 
Associates of Los Angeles, California and helped provide 
a conceptual framework for the IRA of the context in which 
an educational product might be developed, field tested, 
and revised, as well as techniques for conducting these 

activities. 

Self Diagnostic Examinations . All examinations were 
presented in a fashion which enabled the students to conduct 
a self-analysis of how well they had reached the objectives 
for each week. Formal discussions were held to discuss and 
critique the test instruments and to enable students to 
clarify key points. 

Individual Conferences . In order to accommodate the wide 
range of backgrounds and interests of the participants, the 
faculty members were readily available throughout for 
individual discussions. These meetings centered upon the 
work in the Institute but also provided an opportunity for 
participants to draw upon identified interests and 
capabilities of faculty members in solving aspects of problems 
related to their current job assignments. ^ 
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Group Dynamics Sessions . Group Dynamics vas explored in the 
opening series of meetings. This technique vas attempted in . 
order to rapidly integrate the group and to provide an oppor- 
tunity for the participants to release any aggressive feelings 
oxf any hostilities vhich, even though they were indirectly 
related to the Institute, mi^t impede the desired outcomes 
of the educational experience. The major objective for the 
three meetings was to coalesce the group rapidly into a 
working unit in order to focus them upon the training worx at 

hand. 

Traditional Appro aches . The assumption that individuals 
learn in many different ways and from many forms of media 
provided the rationale for including some time-proven approaches 
to teaching, including: lectures, discussion groups, individual . 

study sessions, and films, A synthesized typical week of 
activities in the Institute is provided in Figure 2. 

The "typical" week of activity provides some idea of the variety of 
instructional configurations and approaches that were utilized during the 
Institute. Imbedded in the context of each of the three weeks was a 
group of teaching objectives that were designed to help reach the overall 
Institute objectives outlined earlier. Some of these objectives are 
provided below in order to aid the reader in obtaining a clearer insight 
into the range of behaviors that the faculty felt were worthy of 
exploration in this developmental version of the Institute. 



First Week 



At the end of the first week the students were able to: 
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Identify and discuss the conceptual construct of SPIa- 
Compare and contrast General measurement techniques. 



Describe the processes involved in systematic instructional decision 
making. 



k. 



Critique an initial draft of an interview protocol and sucgoot areas 
for revision. 



5 » 



Evaluate a "live" data collection interview and provide constructive 
criticism for improvement of the sampler s techniques. 



6 Describe introductory constructs in educational measurement: scale, 

’ standard scores, assessment of achievement vs. chanec. 



7, 'Utilise interview protocols at, a beginner s level through repeated 
practice sessions. 



8. Relate the role of the IRA to the overall operation of ERIE 

9. 



r^r-Mnfrui-h co-rectly betveen vritten objectives representing the 
cigSuve, aS;ctiveI and psychomotor domains of pupil behavior. 



10 . Define and discuss the terms reliability, content validity, construct 
validity, control of error, and absolute scores. 



n Achieve scores on'.an attitude measure vhich reflect a 

attUude toward objectives vhich are both behavioral and important. 



18 . Analyze the processes and/or products of the key stages in the 
development of validated instructional ina..orials. 



13. Describe a rationale (Stake's) for evaluating education products and 
programs. 



Ik. Identify five different observational instruments. 



15. Outline the key Issues in subjective and objective evaluation 
procedures. 
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Second Veek . . ’ ■■ ■ 

# 

At the end of the second veek the sttvionts were able to: 

1. Specify the method of sccilin^; paired comparisons. 

2e Analyze correctly and so2.vo a paired coniiiarisons problem. 

• 3. Contrast the evaluation perspectives of the experimenzaliSu, 

the counselor; and the accreartation agency. 

k. Identify the ten categories of the Flanders observational system. 

5. Apply the above system in a preliminary fashion to a video-tape 

simulation and a fili:i of classrooms in progress and eva2.uate the 
intyractions observed. / 

6. Relate scaling methods to SPIF# recuirements for the quantification 
of complex objocts. 

• 7. Conduct interviews and participate in self-evaluation oy viewing 
video-tape recordings of each interview. 

8. Identify pertinent research studies. relating to interviewing and 
■ ■ observational methodology. 

9. Describe the documentary evidence that might accompany a validawCd 
instructional product. 

10. Compare and contrast behaviors for each level of the cognitive ana 
affective domains. 

11. Identify important techniques for conaucting interviews. 

12. Identify and construct student and class performance objectives . 
using both quantitative and qualitative standards. 

• • • 

13. Compare and contrast the methoc*^s for collecting product information 
in the field of educational technology. 

lU. Describe the methods whereby the, overhead projector will be 
treated as a "cLass" in the I2PI3 system. 
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Third Week 



At the end of the third veek the students ’.?ere able to: 

1. Describe the concept of contingency in relation to the EPI3 research 
plan and in tcrriis of correlational methodology. 

2. Improve in the use of ohc SP3TdI interviev; protocol, through aadit:.onal 
practice in use and by vieving video-tapes of thei.iselves conducting 
interviews. 



1 3 , Idontii'y class! ions in socionictry and perform entry level 

tasks in applying nociometric tcchnicues. • . 

4. Compare and contrast nominal, ordinal, interval, and ratio 
scales. 

» t 

* * I ' 

% Analyze five unobtruoivo measures and their relationship to 
eKoerimcntal design and EPIli. 

6. ■ Distiagiiioh between, equivalent practice, analogous practice, and 
■urcivcjiiisitc tasks i’er puj)!! activities anc^ write learning 
activities for each category. 

7 *. V^rite ''pcrceivcd-nur^.'Ose-activitlcs' to include exhortation, 
deduction, induction, and extrinsic reward. 

, 8 . Identify the major categories and products current’!:’ *• for 

coniputer-assisted learning and hov 3PI2 might treat t:...;.. -s a 

product class. 

% 

9 . Describe a set of variables that might be required by any user of 
* the EPII3 system. 

10. Identify ways in which EPIii might utilize a computer tiJAC- sharing 
system. 

11. Synthesize the Tiiajor .research efforts, techniques, and approaches 
to information sampling. 

12. AnaDyze and revise bho objectives for the three week institute. 
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The preceding objectives were attained by most participants during' the 
three weeks. The faculty members were well aware that a variety of 
performance levels would be present because of the wide range of entry 
behaviors and the nature of the objectives. However, the need to 
explore many objectives was important in order to test the initial 
conceptual construct of what comprised potential IRA behaviors. The 
feedback provided by the students and the continuing review of the 
instruction helped, of course, to identify certain objectives that would 
be more valuable than others in creating the final training for an IRA. 



At the culiaination of the second and third weeks of instruction, lengthy 
oral and written critiques of the Institute were requested from the 
participants. This activity provided the participants with a formal 
opportunity to direct their critical appraisal capabilities «o specific 
instinct ional events and sequences of such events. Tne faculty ^roup 
undertook a similar effort. Assessment of the ratings by both the parti- 
cipants and the faculty indicated that this initial Institute was 
considered successful in meeting its overall objectives. However, a 
clear specification was made of numerous activities that should be 
repeated or deleted from any future training effort. 

A few of the general categories for continuing review are: 

. A resolve that as a clear identification evolved oi how the 
JM might operate most successfully in the field, this 
knowledge would be incorporated in the training procedures. 

. Directly related to the preceding concern was one which 
raised the question of how well the role of an IRA could 
be filled by an individual at less than a professional 
level. Perhaps, following the field test of protocols and 
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proceduros by professionals, other less hignly trained 
personnel could effectively undertake the data collection 
tasks. This concern is a tvo-fold probleia: How well can 

even professionals function as IRA.S when they may be 
restricted in terms of time and Job objectives by their 
career assignemtns; and, is it efficient to utilize highly 
qualified professionals in this role on a full time basis, 
once the methods and procedures are identified? A possible 
solution might be to have the professional manage a group 
of less qualified opinion samplers in his particular 
geographic area. 

Possibly there could be an even greater use of the video 
tape recorder in aiding IRAs to develop their interview 
techniques, and also an increase in the use of active 
responding instructional sequences in the training. 

Perhaps shorter and more intensive training which might 
require less time away from professional assignments. 

Consideration of increased opportunities for self-diagnosis 
in dealing with Institute objectives, based upon the 
rationale that professionals are reluctant to expose their 
lack of ability to other practicing professionals. 



During the fall months the Institute participants will obtain producer user 
data utilizing a version of the interview protocol which they helped to 
revise; They will conduct at least three interviews with self-contained 
classroom teachers who are using elementary school science materials. 



Honefully, they will be able to interview teachers at various grade levels, 
Following these activities a two-day follow up session which will involve 



er|c 



- l6l - 



14 



the participants and the Institute faculty members vill be held in 
November. . 

The final Institute critiques proved valuable and further analyses are 
being made to ascertain vhat the participants thought they needed and uhat 
the- faculty felt was vital to carrying out the EPIS-IEA mission. The 
final analysis of uhat knowledge was of most worth, will have to be 
deferred until the assessment of the fall field trials has been made. The 
vital field behaviors will have to be correlated with the initial theore- 
tical construct and the synthesis of the three-week effort. The prototype 
Institute did indeed provide considerable insight regarding what knowledge, 
behaviors, and techniques are of most worth in the training of an 
Information Research Associate. 
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Figure 2 - A CCIIiSDULli; FOR A TYPICAL. WEEK 
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Appendix X: Working Document 



Sampling Design for the Four-State Cooperative Project 



Delimitation of the Universe 

The univerae for the study is defined to include; 

All public elementary schools with formal programs in science in- 
struction during the 1968 school year in the states of New York, 

Pennsylvania, New Jersey, and Delaware. 

A number of the concepts in the statement of delimitation need 
explanation. 

Public elementary schools are here defined as including all those 
operated by and directly financed with state funds. The most prevalent 
form of grouping by grades, about expected to be the 1-6 (or 

K-6) arrangement. In less than of the schools will grades 1-b and 
1-12 arrangements be found. In a very few other elementary 
will find grades 1-3 constituting an elementary school unit. The decision 
to exclude non-public schools derived from the source of initial suppwt 
for EPIB, the U.S. Office of Education, and from anticipated funding from 
state departments of education. 

Kiementary school science was selected as a result of a questionnaire 
survey conducted by the Institute for Educational Development for EPIE, 
utilizing a national sample identified by the American Association of 
School Administrators. Elementary school science was shown to be a 
curriculum area of very great concern to the school systems. 

A formal program in elementary school science is thought to be an 
important sample ^eTequisite. Only those elementary units which formally 
intend to teach science are likely to purchase materials explicitly for 
achieving some intended instructional goal in science. Further, those 
presumably few instances where science is not tau^t formally in the 
elementary school would probably be sufficiently unusual to make it 
difficult to produce warranted generalizations from study data collected 

therein. 

The four-state universe results from the leadership taken by the ^ 
heads of their respective spate departments of eduction. 
earlier in this document, a Title V proposal submitted to the U.S. Office 
of Education for the funding of the Four-State Project proposal was never 
considered since its receipt in Washington coincided with Obsessional 
amendment of the Title V of ESEA.) Though funding throuj^i Title V 
was not possible, the cooperation of the states remains and makes it 
prudent for our restriction of the universe to these four states. 

EPIE considers that these restrictions exclude less than ten per- 
cent of the hypothetically complete universe. Since the Depar^nt of 
Education census statistics for the 1968 school year will not be 
able until 1969 , a more precise estimate will have to wait until the 
requisite confirming data become available. 
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The Sample 



A three-stage sampling scheme vill be followed in the selection of 
schools. The first stage involves the selection of a ten-percent sample 
of the elementary school districts defined as eligible for the purposes 
of this study. 

Clustering by school district is being employed in preference to a 
single-stage sample of students or a two-stage sample wherein students 
would be sub-sampled from each of the eligible elementary schools. One 
reason for this decision is the necessity of employing field research 
agents at each of the schools from which students and teachers will be 
included, in order to insure accuracy of the subsampling process within 
the school and to insure a high rate of inventory and/or interview comple- 
tions. For this reason EPIE thinks it desirable to select relatively 
large numbers of cases from each of the districts falling into the 
sample. It is realized of course, that fewer districts and schools in 
the sample will increase the ease of gathering data — but will correspond- 
ingly increase the sampling variance of estimates pertaining to the entire 
universe. However, the sampling unit will be sufficiently large as to 
help reduce the error, and, balancing the pressure toward a small number 
of sampling units against the pressure toward minimizing errors of esti- 
imte, EPIE feels a first-stage ten-percent sample to be close to optimal. 

A number of arbitrary decisions will undoubtedly be involved in the 
formation of primary sampling units. It will be difficult to determine 
whether schools which deviate from the sample *s descriptors should be 
Included in the secondary sao^le. It is hoped that these will run at 
less than five percent and therefore not markedly bias the results. 

Each eligible district will be allocated to one of ten strata on 
the basis of an index reflecting size, and financial and urban-rural 
characteristics. Values with respect to each of these three variables 
■yill then be calculated for each of the school districts and will be 
used to rank each eligible district independently. O:# three strata 
will be defined as: 

I. Upper ten percent in average teacher salary and per-pupil 
expenditure 

II. Middle thirty-four percent in average teacher salary and 
per-pupil expenditure 

III. Lower ten percent in average teacher salary and per-pupil 
expenditure 

Within each of these three strata the districts will then be 
assigned to three groups on the basis of their location — urban, 
suburban, or rural. Within each of the resulting nine substrata the 
districts will be ordered with respect to a measure of size, and the 
districts will be sampled systematically with equal probability pro- 
portional to the measure of size. In the case of several strata it 
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Mill very likely be necessary to sample vith equal probability a sub- 
stratum composed of the smallest districts in order to avoid intricate 
velg^ting adjustments at the tabulation stage of the field studies. 

The measure of size viU be the number of students enrolled in 
the district in 1967-8 as shown on state department of education records. 
Wherever possible^ these same records will be used to determine the 
pupil, expenditure and average teacher salary figures,* 

The number of schools selected from a given stratum will be 
arbitrary. The numbers will be arrived at by trying to balance the 
needs of the anticipated district-by-district analyses against the 
need to avoid a ponderous system of weights. 

As was mentioned^ differential sampling rates within strata will 
be necessary because one of the chief objectives of the study is to 
estimate the parameters pertaining to the product performance of ele- 
mentary science curriculum materials in a wide range of Instructional 
settings. To assure a sufficiently stable data base where the sample 
design delimitations may severely restrict the population parameters^ 
a disproportionately larger sample will of necessity result within 
these strata* 

For any given school system twe subsampling rate for schools (and 
within the school the subsampling rate for students) will be the ratio 
of the overall sampling rate for the school system's stratum to the 
school system's probability of being selected. It is believed this 
procedure will compensate for any disproportionality between the measure 
of size which is assigned to a system (or a school) at the primary stage 
of selection and the actual number of students found to be eligible for 
Inclusion in the field study. It is anticipated that the assigned 
measures of size will be imperfectly related to the actual size because 
systems, schools, and student bodies will vary considerably as to the 
amount of change which will have occurred between collection of state 
department basic data and the start of the study, 

A Final Word on Sample Estimates To make sample estimates two 
types of weigpiting are thought to be necessary* The questionnaire 
returns, completed interviews, and test results furnished by systems 
and schools within systems with probabilities less than the sampling 
rate for their strata will have to be weighted so as to bring the 
probability of such a school, or such a system, up to the stratum 
level, *niese are most likely to occur in three of the nine substrata, 
as shown in the table on the next page: 



*In those instances where the figures are not evailable from the state 
department of education the school districts will be queried directly 
for these data. 
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Many of the schools in those strata vill be sampled at a lower 
rate which will be compensated for by weighting their returns to bring 
their probability up to the probability of the overall sampling rate of 
their respective strata. This is quite speculative Judgment at this 
writing but it constitutes a reasonable guess. 

Since the parameters of many schools and individual respondents 
within them who will fail to return questionnaires or choose not to 
participate in the study will be unknown, it will not be possible to 
make a precise estimate of the full population parameters. It may be 
possible only to report the departures in normality from the design as 
they are observed, make compensatory adjustments as feasible, and accord- 
ingly employ more conservative generalizations from the findings. 

Field Rtsearch_AssoclHte^ 

A field research associate will be assigned to each of the districts. 
His first task will be to collect the previously cited data from Vne 
state departments of education or the local schoo3 districts. This in- 
foruBtion will be necessary for the establishment of an overall 
rate for the field study. The next task of the field associate will ^ 
to assemble a list of the schools who meet the criteria for eligibility 
detailed earlier in this section. The Institute's senior staff will 
then identify a random ten-percent sample of target schools. 

The cooperation of the officials at each of these schools will 
then be sought by the field representative to confirm the school's 
eligibility and its willingness to participate in the field study. 

In-depth Study 

At sixty of the schools in the sample (twenty each in New York and 
Pennsylvania, fifteen in New Jersey, and five in Delaware), all pupils 
and teachers will be included when the probability of the school s being 
included is either less than or equal to the student sampling rate set for 
the sQhool^stra*^. These schools .will . 9 erye as complete coverage schools 
wherein all instruments and techniques to be employed throughout the survey 
will be administered. 





Appendix J: from Bie PP Forum. S^pioiber^ I967 



a peneareh rationaie for epie 



Robert E. Stake 




Over 95% of all research in education concentrates on indi- 
vidual differences among students. Without contesting the 
importance of those differences/ EPIE has adopted a ratio- 
nale for research concentrated upon differences among 
products. In many ways, product rt'search is similar to stu- 
dent research, but there are important differences between 
the two. Some of these similarities and differences, along 
with the outline of a general research plan, will be spelled 
out in this article. ‘ 



As explained elsewhere in this issue of The EPIE Forum, EPIE 
will examine instructional products of many kinds. Some 
products will fall within the so-called "hardware" category; 
projectors, CAI-computers, tape recorders, and so forth. 
Some products will be the so-called "software" variety; 
textbooks, teacher's guides, films, encyclopedias. Some day, 
a few of the "products" viewed by EPIE may be more tech- 
nique than product; team teaching, flexible scheduling, 
BSCS biology. A research rationale for EPIE has had to be 
drawn generally enough to permit the description and judg- 
ment of a large variety of materials and techniques. 

It might be easier for all concerned if all important usage in- 
formation and instructional products were free of personal 
judgment-easier, but less useful. The physical specifica- 
tions and cost of these products can be documented objec- 
tively, but not their ease-of-use nor their, developers' philo- 
sophical orientation. EPIE's researchers will have no choice 
but to deal with subjective judgments. Their research de- 
sign, however, will be one which attempts to deal with sub- 
jective judgments in an objective, standardized, and forth- 
right way. 

Every product can, of course, be described in a variety of 
ways, and comparisons among products can be made on 
many different grounds. Two dictionaries, for example, may 
differ as to number of words defined, size of lype, durability 
of binding, and attractiveness of illustrations. They may 
differ, too, in less tangible matters, such as the thoroughness 
of definitions or the sanctity in which formal grammar is 
held. One dictionary is likely to be better for some purposes, 
another for other purposes. It will be the responsibility of 
the reseaicher to describe the dictionaries as fully as he can, 
then to indicate the conditions under which he knows or 
suspects that individual dictionaries will do a good job (and, 
sometimes, which dictionary will do a better job). 

But what is a good job? People have different ideas of what 
a product should do— just ask them. One teacher wants a 
social studies textbook to emphasize human dignity. An- 
other wants it to emphasize the increasing cosmopolitanism 
of our people. One administrator wants a book inoffensive 
to each of the neighborhoods in his pluralistic school dis- 
trict. One wants it to shy away from topics that will hasten 
its obsolescence. One parent will only believe in the book if 
it acknowledges that his minority group is subjected to dep- 
rivation and humiliation. Another parent wants the book to 
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cover all the social science a youth needs to gain admittance 
to college. Consciously and unconsciously, people have dif- 
ferent expectatioi.s of the products they use. 

The EPiE research rationale has no stronger commitment 
than the commitment to record and to honor this diversity 
of values. No product evaluation can be complete without a 
survey of the preferences and priorities of the many groups 
of people who use the product, or who may benefit or be in- 
jured by it. 

But this commitment does not preclude forthright state- 
ments of relative values. A moment ago, I indicated that one 
dictionary-to mention one kind of product-may be better 
for one use, another for another use. It is possible- that some 
products will be ideally suited (or, at least, better suited than 
any competing product) for all conceivable uses. But it is 
unlikely. Frequently, a user will recommend a product for 
use with only certain children, for only certain courses, for 
only certain teachers, for only certain times, for only certain 
educational objectives. Occasionally, an educator will spot 
inconsistencies in the way an author develops a lesson or in 
the way a director organizes a film-strip. Occasionally, a 
teacher will spot a substantial bonus of "implicit learnings" 
in a routine workbook exercise. Such flaws and bonuses will 
be more important in one classroom than another. Product 
information from EPIE should reveal both the "target" uses 
and the "out-of-bounds" uses of the product-as seen by the 
producers and the users of the product. 

Let me summarize what I have said so far about EPlE's re- 
search rationale. EPIE will describe products available for 
use in the classroom. Included in the descriptions will be in- 
formation about user satisfaction and dissatisfaction. Many 
different product dimensions and conditions-of-use will be 
considered. Merit will not be presumed to lie on a single uni- 
dimensional scale of value, but will be attached to the fea- 
tures and functions that are important to different users. 
The value of an educational tool is complex. EPIE will never 
engage in simple, universal comparisons. 

rvHvarvh rationnle 

THE USEFULNESS OF PRODUCT INFORMATION 
No one doubts that educators need to exchange information 
among themselves about instructional tools and techniques. 
Yet there is apprehension among educators (and certainly 
among producers) about organized efforts to obtain that in- 
formation-and with good reason. The hazards of prejudice 
are no less than the hazards of ignorance. 

It is obvious to the supporters o. -PIE that the need for in- 
formation justifies the risk of prejudice, i.e., the possibility 
of encouraging an occasional unwarranted innovation or 
maintaining some out-dated standards. The risk can be kept 
small; but the need for information cannot be made small, 
for it grows out of the imperative need for rational decision- 
making. If educators want to offer high quality programs 
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they must know the merits of alternative materials available 
to them. EPIE's research aim is to provide the curriculum •- 
organizer and the instructor with information that facili- ‘ 
tates their planning and increases their professional compe- • 
tence. 

Educators are not *!ene in their need for knowledge about • 
what works in the classroom. The lay public, too, has such ^ 
a need. Though it is unreasonable to expect either the man 
on the street or the community leader to understand the i 
technical characteristics of curricular materials, it is essen- 
tial for the wellbeing of every school system that commu- • 
nity members have access to background information 
against which to scrutinize major curriculum decisions. 
Much product evaluation information should reach the 
community directly from teachers and school administra-- 
tors, but additional reliable information should be available 
as well. Some of it already is. More is needed. In many com- 
munities, EPIE publications will rightfully become at least 
occasional reading for school board members, PTA commit- 
tees, interested parents, and newspaper editors. 

Quite apart from the need for facts upon which rational de- 
cisions can be based is the need for a general skepticism 
about results. The proof of the pudding is in the eating; the 
proof of the teaching is in the learning. Too often, books,’ 
machines, leaching aids, and tools are bought because of 
their physical attractiveness, the prestige of their producer, 
or the intrinsic quality of their components; too seld6m are 
they bought on the basis of results-the results of a step-by- 
step study by potential users or of a standardized inspection 
by disinterested parties. EPIE intends to be a disinterested ’ 
informer, describing in detail facts that have been revealed • 
in laboratory and classroom testing, and emphasizing the 
satisfaction, dissatisfaction and conditional-acceptance of 
users from community to community. And in producing 
these facts EPIE hopes to whet our appetites for more facts, 
and to bolster professional skepticism about results. 

It is reasonable lo suppose that EPlE's influence on instruc- 
tional equipment and materials will tend not only toward 
better quality but also toward greater diversity. New prod- 
ucts can be expected when it is better understood that one 
arithmetic series does not serve all children well, that one 
model of overhead projector does not fit all teaching needs, 
and that one kind of video tape does not perform best under - 
all conditions-of-use. From East to West, from downtown to ^ 
suburb, the conditions of education are not uniform— and in 
a pluralistic society, with heterogeneous children, they 
should not be. Producers of educational products should de- 
sign special-purpose materials whenever they— and re- 
searchers and teachers— identify needs for such materials. 
Because reports from EPIE will attempt to identify the lim- 
its of general use and the potential for special uses, these re- 
ports should contribute to the diversification of educational 
tools. 



n 0‘ TRODUCT INIOF.MATION 

lat does a teacher need to know about a product? The 
iwer will vary from product to product, but one fact is 
tain: a teacher needs to know lots of different things. To 
adequately informed he needs to know the physical prop- 
ies of the product, the purposes for which it was in- 
Lded, the conditions under which it has been used, the 
ual results of its use, and the judgments of users and ob- 
vers. Since each of these types of information may in- 
[ve dozens of variables and hundreds of observations of 
rformance, some device is needed to organize data in a 
ible form. The device EPIE has chosen to use-and, in a 
\se, the embodiment of its research rationale— is the mat- 
illustrated in Figure 1. The cells of this matrix are the 
igeon holes" of EPIE's desk, the mail boxes in its post 
ice— spaces for the storage of information and reminders 
all the different types of communication that will be en- 
antered. 



For a complete up-to-date description of a product, a large 
amount of information will continually be entered and 
stored in each of the nine major cells, each of the 63 subcells 
indicated by dashed lines, and each of many sub-subcells 
(not shown). To cope with this deluge of information EPIE 
plans to classify some data as Intents, some as Observations, 
and some as Judgments. Actually, EPIE will solicit most of 
the data, and the matrix will also serve to guide the develop- 
ment of questionnaires, tests, observation schedules, con- 
tent analysis check lists, criteria for entering new informa- 
tion and for extracting outdated information, and other 
evaluation instruments. 

To begin with, EPIE will describe a product in terms of its 
own rationale (as explicated by its developer or implied by 
its "content" and "recommended uses") and in terms of its 
formal specifications, physical properties, and market infor- 
m ition. The product should also be seen against a back- 
ground of standards accepted by school people for products 
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^re 1. Matrices to guide the collection of evaluation information about an educational product. 




of its type, and it may be EPIE's job to assemble the material 
needed to clarify that background. Elaborated statements of 
these three kinds, i.e., rationale, properties, and standards, 
are represented by the top three cells in Figure 1. 

The other six major cells in Figure 1 will be used to describe 
a product's use. Subdivisions of these cells sort out bits of 
usage information as to type and source. In the middle row 
of cells, the matrix calls for data on the Conditions of 
Product Use “data that will ultimately reveal the circum- 
stances under which the product succeeds or fails. EPIE will 
first note how the producers, analysts, and users expect the 
product to be used— on what children, with what teachers, in 
what classrooms, for what purposes. Then, EPIE will ascer- 
tain actual conditions-of-use 5n a large number of schools. 
To round out the picture EPIE will solicit judgments as to 
the desirability of the various conditions und?r which the 
product is being used. Coupled with knowledge of results. 



the data of this middle row should permit EPIE to circulate 
reliable statements about the generality of a product's appli- 
cation, i.e., whether its results are dependent on special cir- 
cumstances or can be expected just about anywhere it is 
used. 

The cells of the bottom row call for data on Results of Prod- 
uct Use. EPIE's reports will emphasize, as all evaluations 
should, the vis>' !e effects of the product. Outcomes of use 
will be ascertained by testing students, by observing class- 
rooms, by interviewing teachers, and by questioning super- 
visors— not under all conditions-of-use, of course, but with 
as much scope and intensity as budget (and restraint upon 
obtruding on instruction) will allow. As indicated in the de- 
tailed list of Table 1, EPIE clients will want to determine the 
student accomplishments, changes in student attitude, and 
effects upon the staff, the school, and the commurdty that 
may be attributed to use of the product. 
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Figure 2. A representatibn of the processing of the r *oduct information collected by EPIE. 
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cell at the center of the bottom row in Figure 1 repre- 
s observe^ outcome information. The other two cells in 
>ottom row of this graphic rationale represent (on the 
side) infended outcomes and (on the right side) judg- 
ts of merit and shortcoming of the outcomes. Behavior- 
ally minded advocates of "hard-data evaluation" will 
! that these three bottom cells contain, from left to right : 
kvioral objectives, performance data, and systematically- 
ed preference-ratings. Others may prefer more person- 
ed, less standardized statements of goals, outcomes, and 
lies. EPIE hopes to have both hard and soft data to ex- 
nge. 

len an EPIE staff member looks at the complete matrix 
deviated in Figure 1, he will see row upon row of lines 
ther subdividing any subcell shown there. He will expect 
ase many variables (see Table 1 for examples) to describe 
roduct. Each variable has its own place in the fine-detail 
trix, upon which intentions, observations, and judgments 
1 be recorded. Just how the method works will be illus- 
ted in the next section of this article. 

my way of thinking, the elements of any evaluation are 
s of information. Each bit is identified according to di- 
•nsions or characteristics that help to describe the prod- 
t. In the EPIE matrix designed to help the local decision- 
iker evaluate products, each dimension or characteristic is 
jigned a row. Each source of information is assigned a 
lumn. A bit of information, then, has its own sub-subcell, 
|uared off by row and column, identified by type and source 
information. 

ratumuiv 

LKIRCr^.' • •! rnODUCT l\’FORM.-\TION 

fhen an EPIE staff member looks at the matrix he also sees 
ilumn after column aligned for data from different sources, 
ith a separate column for each person or instrument col- 
tting the information. Most of this information is expected 
) come from three major sources: producers, analysts, and 
lers. 

he staff expects that producers will send useful descrip- 
ons of products, their developmental history, ai\d-they 
ape— the results of field testing.* 

Ibviously some producers will be more informative than 
^ers, but EPIE has been assured by many that they will 
rovidc the needed data. 

additional information will come from temporary or ad hoc 
roups of experts, each asked to examine specific features 

The term "producer" here inctudee distributors, manufacturers, 
esigners, authors, publishers, R&D Center personnel, and any- 
ne who has been instrumental in making a product available to 
urchasers. 
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or products. One expert might be asked to compare projec- 
tor lenses, another to search out implicit objectives in a 
"Teacher's Guide," and still another to document the value- 
commitment of a textbook author. These experts will usu- 
ally have access to the product and its paraphernalia; only 
occasionally wUl they participate in the study of its actual 
use. Laboratory testing routines, when they exist, will be 
accessible to analysts appraising equipment, and content 
analysis facilities will be available to those appraising in- 
structional materials. 

The greatest flow of information will probably originate 
in the third source of data, the product users. Using both 
standard forms and open-ended protocols, the cooperating 
users will tell what they used, how and why they used it, 
and with what results. A few schoolrooms may be enlisted 
for studies "in depth", but the usual procedure will be to 
piece together small but overlapping parcels of information 
obtained from many users. Thus, the total sample will be 
more representative, and participating school personnel will 
be less burdened with the task of writing reports. To be 
sure, a plan that does not ask the same questions of every 
user will sacrifice an increment of stability in correlations 
among outcomes and conditions of use. However, the suc- 
cess of sampling methods by social science su^ey workers 
has persuaded EPIE that a more accurate picture can be as- 
sembled from bits and pieces of data from many sources 
than from massive data from a few users. 

To illustrate how the matrix represents information of dif- 
ferent origins and viewpoints, let us consider a sometime 
objective of algebra: the ability to calculate compound in- 
terest. To evaluate an algebra textbook we would assign 
this ability to a row in the information matrix. It would fall 
in the bottom section, along with other results-of-use, and 
toward the top of that section, because it pertains to a "Gain 
in Student Competence," the first "Results" subcategory 
in Table 1. This entire row, then becomes the "ability-to- 
calculate-compound-interest" row, across the matrix from 
"Intents" through "Judgments." What EPIE finds out about 
the author's expectation as to students' ability goes in a sub- 
subcell within the subcell headed "Intents; Producer." Test 
scores from a compound-interest quiz for a class using 
the textbook go within the subcell headed "Observations; 
User." Commnte on the level of these student perform- 
ances, as see. . by an EPIE consultant (in this case, an expert 
in mathematics education), go in the same row, within tne 
subcell headed "Judgments; Analyst." In this fashion, EPIE 
information is classified throughout the matrix. 

It is presumed that each commercial product will have its 
advocates and detractors. It is not EPIE's purpose to tally 
them, or to seek some consensus of opinion about product 
utility. There may be a half dozen points of view about a 
product. EPIE hopes to learn them all, and share these points 
of view with potential users. 
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What is the value of, say, a new geography kit? It is not 
unreasonable to suppose that pupils, parents, administra- 
tors, teachers, geography-education specialists, and geo^a- 
phers will have different answers, and that subgroups with- 
in these groups will have still different ones. Product value 
can be complex. EPIE hopes to sample enough sources of 
judgment to capture and reveal that complexity. 

vpHvnrvh rntiinmii* 

THREE TRADITIONS 

EPIE's research plan draws upon three prominent traditions 
in educational appraisal: a) a guidance and counseling tea- 
dition, b) an experimental tradition, and c) a school accredi- 
tation tradition. The concepts and methods of these tra- 
ditions ‘^ave been merged with such newer methods as 
subject-matter content analysis, behavioral task analysis, 
classroom interaction analysis, opinion surveys, and 
ence scaling to provide a library of techniques for EPIE s 
information gathering and processing. 

Most of the aptitude and achievement testing techniques 
we now use in education were developed for guidance and 
counseling purposes. The primary reason for such testing is 
to describe a student's mental powers with reference to local 
or national norm groups, and to predict the student's future 
standing. In most popular philosophies of education, great 
emphasis has been laid upon individual differences 
learners. Our technical definitions of reliability, validity, 
and errors of measurement and estimate are rooted in the 
concept of differences among individual examinees. Al- 
though not all school counselors choose an empirical base 
of test results for their work almost all test results have 
been designed for the questions that arise in guidance and 
counseling. 

EPIE hopes to borrow from this tradition both in spirit and 
in technique. Much can be concluded about a product s im- 
pact from an analysis of individual student performances, 
and this is important data. But as Cronbach* and others 
have pointed out, the answers to questions about teaching 
practice, curriculum development, and product appraisa 
• will often be new questions, and the methods of research 
will often be new methods, not the traditional psychometric 
ones. Variance among products is not equivalent to variance 
among students-and the former will be EPIE's focus. ?tod- 
uct research needs its own measuring devices, but the tradi- 
tional psychometric tests-paper-and-pencil, objective tests 
* —will be emphasized only when they have obvious content 
validity, i.e., when they directly answer questions pertinent 
to the. selection of instructional products. 



*Cronbach, LI. Course improvement through evaluation. 
Teachers College Record, 1963 

- 17k 



The experimental technique is one of controlling condi- 
tions-of-use" and the manipulation (controlled variation) or 
a main treatment to effect some carefully specified outcome. 
From an experimental study, the researcher hopes to learn 
something about relationships among a few variables. He 
hopes that these relationships do not exist under laboratory . 
conditions alone-that they are independent of such inci- 
dental conditions as who the investigator is, what time or 
day it is, and the like-but he seeks at least to ascertain the 
extent of relationships when those laboratory condittons do 
exist. Findings from experimental studies in education are 
sometimes considered of little practical significance because 



Product research is similar to experimental research in that 
the emphasis is on the relationship between treatment ^d 
outcome. But EPIE's studies will be more naturalisHc. T^e 
conditions of instruction will not be controlled. Massive 
replication and careful analysis will be needed to rule out 
alternative explanation of the results. 

The expectation is not that EPIE will conduct experimental 
studies, but that EPIE studies will reflect the experimental- 
ist's concern for conditions of use and for relevant outcome 
performances. The EPIE studies will be"non-manipulative," 
even to the extent of refraining from an exchange of user 
information about the product and, in fact, from any profes- 
sional discussion with participants while the study is under- 
way. EPIE recognizes that the apparent worth of a product 
can be influenced by the very fact that observations are be- 
ing made— an enormous obstacle to objective research. 

In thel: efforts to measure accurately, psychometritians and 
experimentalists have denigrated the role of personal judg- 
ment. They seldom simply put their questions to the ex^n- 
enced practitioner. Not so the accreditation agencies. Their 
methods of appraisal depend directly on the educator s tal- 
ent for assessing the quality of a program. An evaluation 
plan common to most school accrediting agencies calls for 
self-study by the school staff, followed by visitation by col- 
leagues from similar institutions. Although there is public 
emphasis on tangible assets in the accreditation method, 
there is also substantial concern for any administrative and 
insti jctional conditions that obstruct the teacher. Emphasis 
is also given to important differences among the major sec- 
tors of the curriculum-science, social studies, and humani- 



ties. 

The merit in borrowing from this tradition is obvious to 
EPIE researchers. First, there is the necessity of orienting 
EPIE activity to decisions made by actual users, in contrast 
to decisions made by developers, researchers, or theorists. 
Second, there is the necessity of being attuned to uniqueness 
in the needs of the different subject-matter departments, 
even in evaluating hardware or reference works which may 
appear to be independent of subject-matter usage. Finally, 
there is a wisdom among practitioners too precious to ignore. 



lOugh many are the difficulties of validating that wisdom 
a technical sense. EPIE hopes to contribute to the refine- 
ent of observation apprais^ methods, and to rely heavily 
\ them. 

)r processing data, synthesizing findings, and interpreting 
dationships the talents of the logician, the philosopher of 
dence, and the historian should be added to those of the 
Rychologist, sociologist, and economist. It will often be 
iluable to rely on frames of reference more or less indig- 
nous to these disciplines. 



Perhaps the most general search among the data will be for 
(1) congruence between what was expected of the product 
and what actually occurred and (2) contingency relationships 
between outcomes and conditions-of-use which reveal the 
limits of a product's effectiveness. Automated data-process- 
ing techniques for teasing out congruence and contirtgency 
will not be available in the near future. EPIE will rely on 
researchers and analysts with a broad range of talents and 
diverse methods of inference to bring about some orderly 
confluence of data. 



ABLE 1. Subdivisions of information classes for evaluating educational products 



Subdivision 


Examples of variables in the subdivision 


CONDITIONS OF USE 




Local Circumstances 
Student types 
Teacher type 
Type of school 
Type of community 


(background, aptitude, aspiration, . . .) 
(experience, style, personality, . . .) 

(physical plant, intellectual climate, . . .) 
(support of schools, attitudes, controversy, . . .) 


Curricular Context 
Subject matter coverage 
Instructional aids available 
Concurrent coursework 


(concepts, structure, methods of inquiry, . . .) 
(library, models, maps, equipment, . . .) 
(sequence and time allotment, projects, . . .) 


Classroom Transactions 
Teaching strategies 
Student-Teacher interaction 
Student-Student interaction 
Incentives, grades, etc. 


(discourse, inquiry, assignments, . . .) 
(information, flow, counseling, . . .) 

(social climate, reaction to authority, . . .) 
(motivation, goal orientation, testing, . . .) 


RESULTS OF USE 




Gain in Student Competence 
Knowledge 
Skill 

Incidental learning 


(data, understanding, application, . . .) 
(problem solving, communication, . . .) 
(synthesis, learning sets, side effects, . . .) 


Change in Student Attitude 
Interest 
Commitments 


(opinion, avocation, exploration, . . .) 
(prejudice, aspiration, advocacy, . . .) 


Effects on Staff 
Teacher changes 
Administrative changes 


(insights, revision, grievances, . . .) 
(organizational rearrangements permitted, . . .) 


Oiher Effects 
Institutional effects 
Community effects 


(prestige, solidarity, . . .) 
(controversy, dedication, esprit, . . .) 
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SPCCIAL PROBLEMS 

Though EPIE's primary methodological problem is one of 
relating goals and conditions-of-use to outcomes, other 
problems vie for attention. Many of them are old puzzlers 
among teachers, researchers, and producers of educational 
material. One such problem is the specification of target 
populations. Whom shall we ask our questions? What prod- 
ucts do they want to know about? What schools will par- 
ticipate in the information exchange? What steps should be 
taken to assure that classroom conditions are representa- 
tive? What steps will equate one study with another? Initial 
instructional-setting data will provide partial answers to 
some of these questions, but right decisions must first be 
made to collect the right data. 

Any research study seeks generalization, but studies differ 
as to the level of generalization they seek. A study of one 
teacher's experience tells of the specific. The experience of 
a thousand sampled teachers permits generalization across 
teachers. The desirability of generalization across types of 
pupils, subject matters, regions, and time is also to be con- 
sidered. The "basic research" study in education is usually 
indifferent to (allows generalization across) personnel, sub- 
ject matter, locality, and time. The practitioner's inquiry 
usually calls for minimum generalization, because a pur- 
chase to meet some given need is in the offing. But EPIE will 
have many clients. EPIE's studies will specify the product, 
and search for generalization or limits related to types of 
pupils, teachers, schools, and so forth. 

Another problem is that of comparing products with differ- 
ent purposes. No two instructional aids have identical ob- 
jectives; won't an evaluation be fairer to one than the other? 
The EPIE plan is to ask some questions which will show 
each product in its best light, as well as in lights which are 
best for its competitor. Usually, one product will appear 
tietter under certain limited conditions, poorer elsewhere. A 
generous consideration of conditions should permit th'^ po- 
tential user to decide which objectives, which conditions, 
and which products are most appropriate for his school. 

Still another problem has to do with standards. Most oper- 
ating standards are idiosyncratic and unconscious, serving 
to shape personal preferences, perhaps very consistently, 
but avoiding public exposure. The advocate of this or that 
standard may adhere to still others in his own practice. 
EPIE's purpose is not to show what is popular, but to reveal 
—to expose— the various expectations that exist. To be sure, 
exposure is easier planned than accomplished. Surveying 
every expectation is all but impossible; utterances both per- 
tinent and suitably documented are difficult to find. A thor- 
ough presentation of existing standards is a formidable 
obligation, but a necessary one for a nationwide project. 

A fourth EPIE problem is that of identifying school goals. 
Are products differentially useful depending on school 
goals? Of course they are. Along with other statements of 



opinion and judgment, school goals have a translation prob- 
lem. Each goal has implications for practice, but spelling out 
what practices are consiste.it and what are inconsistent is 
not an easy task. Existing statements of objectives either are 
so general that they leave doubt as to what (if any) "good" 
outcome lies outside their scope or so specific that they 
guarantee incomplete coverage of what the educator be- 
lieves he is and should be accomplishing. In addition/ there 
is the problem of giving meaning to priorities. Scaling meth- 
ods are available, but no satisfactory way of quantifying the 
importance of different objectives has yet been devised. 
Maguire* has recognized that priorities given to academic 
objectives may be seen as indices of the time to spend pur- 
suing them, as indices of the extent to which remediation 
should be carried if the objective is not achieved, or as 
indices of personal commitment and value. How can goals 
be quantified imambiguously? EPIE will try. Hopefully, 
more definitive scales for goal priorities will be a byproduct 
of the EPIE studies. 

r0!He.tu*0*h rtt # i on o I o 

CHECKS .AND BALANCES 

Recently, one producer of educational products said, "I 
don't worry that EPIE will accuse us falsely, but that it will 
damn us with faint praise." Failure to recognize important 
distinctions, on the one hand, making too much of small 
distinctions, on the other-these are perils on either side of 
a narrow path. How can EPIE keep to the straight and nar- 
row? Obviously it is necessary to have a staff immune to 
alliances with or grievances against producers, and to accept 
from producers no support of any kind, e.g., advertising, 
free produ( ’ mples, and so forth. But beyond such a<^ 
ministrative and commercial safeguards, there are research 
checks and balances to invoke. 

One principal safeguard is the deliberate over-inclusion of 
descriptive variables. Redundancy (ask the same question 
several ways) and extended boundaries (ask some periph- 
eral questions, too) will lessen the chance of false cause- 
and-effect reports. A welier of data will make any one find- 
ing less critical. Only if everything goes badly- a very un- 
likely result-would a general condemnetion be appropriate. 
The probability of a general commendation is equally small. 
This may be the danger the producer feared-that mention 
of merit would be so buried in data that, in effect, merit 
would not be acknowledged. But this seems the lesser dw- 
ger. Over-inclusion of variables is one of the safeguards 
EPIE will employ to wjlrd off judgmental bias. 



•Maguire, T. O. Value Components of Teachers' Juf.Smejtts of 
Educational Objectives. University of Illinois, Unpublished Doc* 
toral Dissertation 1967. * 




t is the user— not EPIE— who must sort the descriptions and 
udgments to find the bases for local decisions. Classifying 
»r ranking products as to such factors as cost, reading diffi- 
xilty, or durability, may be useful— but classifying or rank* 
ng them as to overall merit is not EPIE's plan. This restric- 
ion on grand inference will be a second safeguard against 
nisinformation. 

K third safeguard will be a reinforced striving to improve 
he accuracy and precision of measuring devices. When in 
full swing, EPIE expects to make contributions to the tech- 
ftology of educational measurement. The search for new and 
Mtter techniques should be manifested in a healthy skepti- 
cism toward the credibility of its own findings. Estimates of 
confidence in its findings should be apparent in the reports 
af product usefulness. 

^e ultimate validity of any technique is established by out- 
side criteria. We do not anticipate that an outside, hard-data 
criterion will soon be available against which to validate 
^PIE's research activities, but other criteria should be avail- 
able before long. Clinical practice among educational plan- 
ners should be influenced, and studies should show that 
influence. EPIE will be tested in the field. If its inforri..^tion 
is useful, more and more decisions will be shaped by it. 
(Validity is not assured *by use— but nonuse suggests low 
Ivalidity. Knowledge of what is done with EPIE information 
is another check upon the system. 



|These checks are insufficient to guarantee honesty, candor, 
land relevance. Professional educators must also monitor the 
iwork. EPIE is considering the need for an independent board 
|of examiners to aid them. Such a board could draw criticism 
and advice from the professional educator, on the one hand, 
and from the producer, on the other. Chosen by professional 
societies, though not necessarily official representatives of 
them, the members of the board would examine EPIE pro- 
cedures and reports, investigate and endorse justified ap- 
[peals for redress, and publish critical reviews. Space in EPIE 
[publications can be made available for their reactions, but 
other outlets will be needed as well. 



The research plan described here is not a hard and fast plan, 
as any research designer is aware. But no hard and fast plan 
is wanted at this point. The procedures of gathering data, 
[drawing up limitations of product effectiveness, and ex- 
khanging information are expected to evolve, improving as 
|new techniques are tried and as feedback on the system 
iltself is available. Just as EPIE seeks generalizations about 
[products, noting their impact under various conditions of 
[ttse, it also seeks to tmderstand the generality and limita- 
tions of its research plan. This paper has indicated where 
it starts. 



Robert E. Stake, Professor of Educational Psychology at the 
University of Illinois, is Associate Director of ClRCE—the 
Center for Instructional Research and Curriculum Evalua- 
tion. His sound and creative approach to gathering and deal- 
ing with data, his intellectual curiosity about the value of 
"soft data," his uncompromising integrity are revealed in 
this paper. 

Among other activities and positions. Dr. Stake has been 
a Psychometric Fellow at Educational Testing Service, has 
taught at the University of Nebraska, and has been a re- 
search consultant to the U. S. Office of Education, the 
North Carolina Advancement School, and the Cooperative 
Educational Research Laboratory. He edited the highly pro- 
vocative Issue No. 1 AERA Monograph Series on Curriculum 
Evaluation (1966). Perhaps most relevant to EPIE among his 
published works is "The Coun lance of Educational Evalu- 
ation," which recently appeared in Columbia University's 
The Teachers College Record. 
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can epie help 
teachers and students 
^HeU it tike it 
(parti) 



Terry Denny 




The phrase, 'TeU it like it is/* is popular with a segment 
of youth often antagonized by and antagonistic to educa- 
tion. The intellectual and emotional content of this message 
refi^cts an increasing social commitment to attack the real, 
everyday practical problems of a growing, if not yet a 
great, society. No widespread hue and cry has been taken 
up by youth in the schools for educational evaluation as 
such. Although students' general concern with the quality 
of their instruction is evident, their specific concerns usually 
'amount to some displeasure with a particular group of 
teachers, with the behavior of certain administrative per- 
sonnel, or with their rights to assert themselves in some 
manner or another. 



There is, however, evidence that a growing segment of pro- 
fessional educators charge educational researchers with 
conducting surveys, field studies, controlled comparison 
experiments, and laboratory research which either fail to 
"tell it like it is," or show little promise for influencing on- 
going school practice. Nor are they pleased with their own 
results of conducting do-it-yourself educational product- 
assessment and curriculum-research studies. Out of these 
concerns have grown several groups, centers, consortia, 
which are national in scope and ambitious in purpose. One 
wonders how these will succeed where so many have failed. 

Paul Mort (1), for example, over a period of thirty years, 
conducted exhaustive studies of how an educational idea 
gets into practice in school districts. He devoted a profes- 
sional lifetime and directed the work of several other 
people in an attempt to attack a real problem in the real 
world of the school decision-makers. Researchers interested 
in extending or replicating his work and practitioners in- 
terested in solving their school problems find the common- 
sense constructions reported in his research nearly impos- 
sible to apply. 

There are obvious differences of intent for the educational 
researcher who seeks to advance today's speculation to- 
ward tomorrow's knowledge and the school practitioner 
who needs to make today's decision. The researcher specu- 
lates about tentative research findings; the user wants 
evaluative statements on which to base judgments. The 
user's professional day is carried on a continuous stream 
of decision-questions: Is this year's reading series working 
better than last year's? Shall we adopt this science textbook 
or that? Which overhead projector shall we buy? Which of 
our current curricular m(iterials shall we continue to use, 
discontinue, amend? He is not satisfied with the local infor- 
mation he has at his dispo.'^al. He finds it too informal, in- 
complete, perhaps inconsistent, and turns to promotional 
literature he has received from publishers and other pro- 
ducers and perhaps to educational research journals for 
published outcomes of curriculum-research studies. 

Reading the producers' promotional literature often does 
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lip him evaluate the absolute or relative performance 
! products he is interested in for use in his schools, 
he digs into the research journals for help he emerges 
no fewer questions and may feel that the researchers 
tting him down by not attacking his real problems. To 
rustration add the likelihood that he is lacking neces- 
time, money, and staff competence to conduct exten- 
>roduct-ccmparison studies. At this point, after some 
Bsion with a fellow teacher, curriculum coordinator, 
nistrator, or salesman, he decides to go it alone on 
tuitive basis. 

fabricated decision-making cycle pleases no one, yet 
Sts. Why? I think the educational decision-maker deals 
his professional world practically. He is inclined to 
f each event as a self-contained problem. He has grown 
Btomed to working with, perhaps expecting, incom- 
[ product information on which to make decisions, 
(lems arise, decisions are made, things get better. His 
cs are often ingenuous, his strategy unformulat^. 
e is learned that can be communicated. These devilish 
I between what decision-makers want, what research- 
do, and what might change the way things are have 
i treated searchingly in separate works by Robert Stake 
D. L. Clark and E. G. Cuba (3), and Matthew Miles (4). 

1 EPIE system could provide the practitioner with the 
tidimensional information which is needed to trarisform 
current hunches into rational hypotheses, to enrich his 
xhial viewpoint with reliable product performance in- 
nation based on a nationwide sample. Notwithstanding 
risk of a tautology it seems important to note that the 
E assessment model itself has demonstrated logical con- 
ence to date, but awaits the test of empirical analysis, 
t proposed pilot study of elementary school science 
ducts in four Eastern states will provide EPIE with its 
ial empirical feedback. These first steps toward establish- 
an information system to provide dependable product 
formance will, it is hoped, stretch into full strides toward 
widing solid product information to those who must 
ir the responsibility of making educational decisions. 

e first general guidelines of this emerging system have 
tn drawn elsewhere in this issue of The EPIE Forum, 
Stake's paper is crucial to understanding the EPIE 
proach to describing fully and fully judging a product's 
rformance. The model includes logical and empirical 
lalyses of the intentions and the outcomes of the pro- 
icers and users of educational products. Doing an effective 
> of telling it like it is begins with descriptive compari- 
ns of the aims intended by the writer, producer, manu- 
cturer of a product with the aims intended by the teacher, 
irner, administrator, supervisor, user of that product. 

ating goals, objectives, aims and the like meaningfully is 
fburdensome task. Percy Bridgeman (5) championed the 



course of operationalism in his classic The Way Things 
Are. His aim was not specifically to influem e educational 
researchers, evaluators, or decision-makers. Rather he 
sought to promote an approach to grounding knowledge in 
precise language with empirical references. The popularity 
of the standard assertion of the oositivist that "whatever 
exists, exists in some amount and that amount can be 
measured" can be traced to Bridge nan's efforts. It is diffi- 
cult to find a well-marked trail from his early work to the 
burst of interest in stating objectives operctionally. What 
is observable is the historical debt we owe to Bridgem^n's 
work. Consider for example the recent work of Mager (6) 
and Gagne (7). Gagne has asserted that the "central focus 
for change in educational practice during the past decade 
has been curriculum." These curriculum innovations fly 
on the wings of products designed to carry their unique 
messages into the classroom. He also cites the absence of 
systematic investigations of the effects of introduction of 
new curricula. Gagne feels that educational content is de- 
rivable only from educational objectives. For example, one 
cannot select content as one might an overhead projector, 
science textbook, or social studies materials. Rather the 
content of a curriculum is the operations a learner acquires 
under a single set of specified learning conditions. Bridge- 
man suggested that often answers were prematurely sought 
before the right question had been asked. Similarly, Gagne 
has suggested that inferring curricular objectives is pri- 
marily "a matter of asking the question of each task. 
While some views of curricular evaluation have the de- 
ceptive appearance of being simple, I tend to concur with 
Ahmann that the task is better understood as horrendous. 
But analyses of intents must be undertaken to give mean- 
ing to the results of using educational products. Ei-!E 
must attempt to navigate a true evaluation course between 
the attractive Charybdis of using well-tested but inade- 
quate methodologies and the beckoning Scylla of employ- 
ing untested but seemingly more appropriate procedures. 

The recent flurry of announcements about marriage'? of 
"hardware" and "software" partners— electronics firms and 
publishing houses-for the purpose of producing new in- 
structional materials has increa'-ed further the interest an? 
concern of educational decision-makers. Romances are 
nearly always heady for the participants but can be stress- 
ful for other interested parties who are often called upon 
to suffer, support, and assuage the couple and their issue 
should the plans go awry. We will have to have the fore- 
bearance, faith, and good judgment to await the first mar- 
ketable products of these mergers, 

I would like to discuss a currently minor but predictably 
greater problem which will visit educational decision- 
makers and EPIE. I refer to the likelihood of a great number 
of producers marketing a still greater number of products, 
the need for which may be questiona'^le. A great number of 
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product models could be fully described by EPIE and the 
prior question as to the real need for any models of the 
product would remain unanswered. EPIE will have to exer- 
cise considerable care in maintaining current product per- 
formance records lest it present outdated information. Fur- 
thermore there is the risk that effective educationa,' prod- 
ucts not profiled by EPIE could be viewed with disfavor by 
EPlE's clientele. EPIE must have built-in safeguards against 
such misuses. 

At this time, EPIE can but acknowledge its responsibility 
to be on guard in these matters. The verbal beau geste 
followed by inaction is ail too frequently observed in the 
educational world. EPIE's responsibility is not met by the 
announcement of such intentions but the likelihood of its 
successfully completing its mission is increased by its 
awareness and willingness to meet the issues head on. 

(The conclusion of this article will be in the October issue 
o(The EPIE Forum.) 



FroKt 1960 to September 1967, when he became Senior Staff 
Associate and Coordinator of Field Research for EPIE, 
Terry Denny had been Instructor, then Assistant Professor, 
and then Associate Professor cf Education and Psychology 
at Purdue University. Earlier, Dr. Denny was an elementary 
school teacher in the Livonia, Michijan, Public Schools and 
has had Graduate Assistantships at the University of Il- 
linois, where he was awarded his Ed.D. in Educational Psy- 
chology in 1962. 

He has written scores of papers, abstracts, essays, and 
reviews for a variety of journa' and other publications, 
concentrating on the areas of reaaing, of anxiety-creativity, 
and of achievement of religious values (particularly in 
connection with the Study of Catholic Education at the 
University of Notre Dame). He has served as a reading 
consultant to American Book Company and has received 
research grants for a reading study from the Purdue Re- 
search Foundation and Scott, Foresman & Company. An 
educational researcher committed to designing sound tech- 
niques and mechanisms for EPtZ's developing system. Dr. 
Denny begins in this issue of The EPIE Forum an account 
of how EPIE will gather user information} the conclusion 
will be published in the October issue. 
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(PART II) 

CAN EPIE HELP TEACHERS AND 
STUDENTS «TELL IT LIKE IT IS*? 

By Terry Denny 



Educational materials are currently purchased on every 
criterion except the results of their use in the classroom, 
which is the principal source of information about the effec- 
tiveness of a product. Robert Stake has called this source 
the "condition of use" in his September EPIE Forum article, 
"A i^esearch Rationale for EPIE." The "conditions of use" 
subsume the teaching transactions, and ^product's outcomes 
are thv*. results of its use. Classroom observation schedules, 
such as Flander's Interaction Analysis technique (1960), are 
useful approaches to charting instn’ctional trans. actions. 
Outcomes can be seen to involve students- teachers, cur- 
riculum supervisors, administrators, parents, the commu- 
nity, as well as certain features of the physical environment. 
Information can be gathered by check-lists, inventories, 
questionnaires, tests, and by classroom observation. What- 
ever the particular approach, EPIE v/ill attempt to use non- 
reactive measures to avoid intruding on classroom prac- 
tice wherever possible. 

Most users will be concerned with teachers' and stu'^ents' 
views of the product, with its effects on students' knowledge 
and skills, with its behavior relative to other similar prod- 
ucts, and with its effects on sttu'lents' interests and attitudes. 
So will EPIE. The information will be "hard" and "soft." 
There are many ways to describe and judge a curriculum 
produc'., and the family of educational materials has some 
remarkable specimens. Those who describe all educational 
matters solely in behavioral terms remind us that EPIE must 
keep its descriptions objective, its observations pure. It is 
easy to agree with the aims of the behaviorist and to recog- 
nise the danger of faddism as well. When an either/ or 
propo.'* .ion is made about the use of operationalism in edu- 
cational research and about the value of an educational 
product, I am reminded of Nietzsche's label for false doc- 
trine, "the dogma of immaculate perception." The observer 
interacts with the observed. He changes the instruction by 
his presence, and he is changed by what he experiences in 
the instructional setting. No amount of operationalism will 
make our measures completely nonreactive, our perceptions 
pure. 

The relevance of hard data is open to question, regardless 
of how tidy the research design may have been that pro- 
duced the well-controlled findings. This is not meant to 
deny the validity of the experimental study; quite the con- 
trary, it has enabled much of EPIE's vision. The point is 
not to pit one methodology against another. Rather, we seek 
to find whatever coherence is possible out of them all— 
without resort to ideological struggles. 

Informal evaluation procedures make use of instructors' 
opinions. EPIE will also. We propose to exercise consider- 
able care in our selection of a sample and utilize a struc- 



tured and open-ended interview schedule to harden this 
very useful soft apprrach to gathering information. Simi- 
larly, we have taken another look at an old educational 
evaluator's chestnut, that of pondering the logic of a pro- 
gram. To this end EPIE has encouraged several curriculum 
content analysis groups to develop and test systems for 
studying the content, rationality, assumptions, valu'", 
promises, pedagogical imperatives and required educa- 
tional technology to be found in the content of educational 
materials. These and other analytical and empirical tech- 
niques will be field tested in a series of forthcoming studies 
which will be conducted within a large-sample pilot study. 

niErii.oi sruP'i 

A large cadre of professional field workers will be coordi- 
nated by the Institute in its four-state pilot study. We are' 
limiting our pilot to the study of outcomes related to 
elementary science instruction materials. In addition, the 
use of the overhead projector has been selected as an 
example of educational hardware to be examined. Research 
workers will gather data on the teachers' aims for these 
materials. Teachers' descriptions of intents will be related 
to their methods of using them and to the empirical re- 
sults of their use. Students' intentions will also be explored. 
But, the emphasis will be placed on variances among edu- 
cational products rather than among the learners. The 
principal criterion for product evaluation is its absolute 
or relative effectiveness in accomplishing its aims. This 
judgment may be made by comparing the results of use 
with other competing products, with the users' intended 
outcome, and with producers' avowed goals for the product. 

Ludwig Mies van der Rohe's architectural principle, "less 
is more" seems apropos of this emerging concept of educa- 
tional product evaluation. Relative product performance 
comparison data should reveal that one product is prefer- 
able to another for a particular user's instructional setting. 
The fewer the gray areas of information about product 
performance, the more certain the user can be. 

Other types of outcomes which interest us include the 
implications for teacher education and in-service education 
which certain materials may portend; the effects on student 
and teacher morale; possible contingencies with grading 
schemes and reporting to parents; the implications for 
administration, scheduling, and supervision; the need for 
supplementary materials, tutors, field trips and additional 
facilities; and the effects that products may have on other 
aspects of the curriculum for which they are not formally 
intended. Beyond the outcomes in evidence when formal 
instruction is concluded lie important results of use such 
as the students' application of knowledge, transfer, reten- 
tion and ease of re-leaming. 



.•RUDUC r ASbl SSMENT IN THE bUIOOLb 

The Institute envisions a new cooperative role for school 
districts in curriculum evaluation projects. The innovation 
and research of educational products are not the most ap- 
propriate tasks for school districts to assume, as we see it. 
Rather they must lead in the development of demonstration, 
trial and modification of the products to the local require- 
ments. Consider this view of the schools' role in the light 
of Robert Stake's caveat, "Even in an iriformal way, no 
school can evaluate the impact of its program without 
knowledge of what other schools are doing in pursuit of 
I similar objectives." 

I Criteria vary from stage to stage in curriculum development 
[ throughout the instructional year of a teacher. This should 
I give us pause in considering the appropriateness of a uni- 
[ variate comparison model to assess product performance. 

[ One could select a significant variable, do a comparative 
i study, select another, and so on. What a relatively simple 
matter the task would be! The utility of such comparisons 
is dubious. We must resist demands for simple, univariate 
I comparisons. Products will have many characteristics, and 
! the characteristics will be of unequal importance. The com- 

I posite picture will result from myriad bits and pieces of 

many educational scenes rather than a detailed painting 
of a few. Judging which characteristics to attend to, and 
which programs to use as references are the tasks for EPIE. 
We earnestly call for your help in solving these basic prob- 
lems of product assessment in the schools. 

To the fellow professional who observes that there is 
nothing new under the educational sun, we submit he has 
been in the sun too long. We invite him to step back, read 
any of the recent, provocative works of Tyler, Stake, Gaj,ne, 
Scriven (1967), Gordon (1967), and Morrissett (1966), and 
help EPIE take the next steps forward. The phoenix-like 
resurrection of curriculum research and the emergence of 
new approaches to new problems encourages us that the 
I time is appropriate to find out the way things are and to 
I measure what the teacher and the student intend to have 
[ happen with instructional materials. Obtaining authentic 
\ statements of intent is a new challenge for the educational 

I evaluator. Teachers must be queried, their views system- 

ic atically gathered and processed, and added to students', 
parents', superintendents', and curriculum specialists' views 
E on the mer.<s and the shortcomings of products. These 
groups have important opinions on education. Their views 
are needed to produce a map in sufficient detail to be truly 
[ useful as a guide to decisirn-making. Controlling bias in 
I this kind of information presents a very real problem. How- 

I ever, the need for results of use data makes it worth risk- 

ing the bias that may creep into the system. 



To see educational life in the round, to avoid parochialism 
in •'‘^hodology as well as in interpretation, and to begin 
the exchange of information among educational profes- 
sionals about product outcomes have already convinced 
The EPIE Institute that it is going to be hard work. Cer- 
tainly little of it could be called romantic. If it were, the 
current charismatic leaders of educational movements 
would already be about these tasks. Melville believed no 
one had to sail in whaleboats to find sharks. The voyage to 
a fully operational information exchange about educational 
products may be longer and more treacherous than first 
appeared. Hard work and patience are the tools at hand to 
implement the Institute's rationale. 

A FINAL WORD 

The information related to the results of product use which 
The EPIE Institute will provide is not intended to describe 
school or school system policy. It is intended to describe 
the instructional outcomes and accompanying practices 
which have occurred in the use of the product. The Insti- 
tute's report to its clientele will not provide the curricular 
model, the criteria list, the best product. There is no need 
to ordain a criterion or a method. There is, in fact, much 
reason to avoid assiduously this cultish danger. The very 
idea of some sort of a final ranking of products is outra- 
geous and would of necessity tend to reduce rather than 
facilitate information exchange. 

Rather, the Institute's services will enable the client to 
have a greater quantity of relevant information to guide 
him in his own decision-making regarding the use and 
purchase of educational materials; to have more informa- 
tion on which to base his own research; and to have sound 
information to guide him in determining his future educa- 
tional needs and policies. The user must use EPIE to find 
bases for his local decisions. When reliable, comparable, 
information about educational products and their results 
of use is made available on a nationv/ide basis, we envision 
a significant improvement in the quality and effectiveness 
of educational decision-making about the purchase, use 
and modification(s) of educational products. 

Product profiles which report the results of use will not be 
simply put, easily perused or quickly implementable. A 
stable characteristic of the educational market place is 
change. Considerable change increases complexity. Since 
the form of educational product information will, of neces- 
sity, be complex, it is crucial to design a system which will 
be quickly and persistently collecting, processing and mod- 
ifying information on the results of use. Outdated informa- 
tion could be useless, or misleading. Ideally, today's data 
should be transformed, translated, and transmitted imme- 
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diately. Alas, the world of the classroom does not permit 
the happy state of instantaneous gathering/ processing/ 
and reporting of information. But EPIE does not promise 
happiness. Even if the problem of data-lag were licked/ 
accurate/ complex/ up-to-date information will make some 
users quite unhappy. 

But it is also likely that continued ignorance will make such 
users no more comfortable. If we cannot get along with 
complex information about a complex problem/ we cer- 
tainly shall not get along with simple solutions to them 
unless we are compromisers at heart. When we attest to 
our heartfelt dissatisfaction with the lack of useful per- 
formance information on educational products/ we can be 
sure that some unhappiness will result from our attempts 
to change things. That's the way things are. Harry Golden 
has written that unhappiness is a cobra which will not 
strike you unless you startle it. Let's risk it in trying to 
help teachers and students to "tell it like it is." 
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This article and the article which appeared in the September 
EPIE Forum were written by Dr. Denny in his role as an 
EPIE consultant prior to his appointment to the Institute as 
Senior Staff Associate and Coordinator of Field Research. 
From 1960 to this September ' ” spent his time as an educa- 
tional psychologist in the Departments of Education and 
Psychology of Purdue University. He received his doctorate 
in elementary education from the UnivB.i>ity of Illinois in 
1962. He now spends his days at the Institute hard at v'ork 
translating consultation rhetoric into plans for field practice. 
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